I'm having issues dealing with a server which would close my FTP control connection in spite of me sending keep alive requests with ftpClient.setControlKeepAliveTimeout(CONTROL_CONNECTION_KEEP_ALIVE_INTERVAL_SECONDS);
Also sometimes it just dies out of SocketException and not the normal FTPConnectionClosedException. All in all, FTP is a very dodgy protocol I'm using a lot and each server I connect requires a bit of tweaking, however this one is giving a very hard time.
I know there's a million things I could be doing wrong, my question is, is there some solution with retries already implemented on FTP in case you lose the control connection, since is not something that should be shocking (proxies / firewalls sometimes just randomly lose your connection). Or is there some more elegant way of solving this problem.
I have something like this.
public void store(File fileToUpload) throws IOException, InterruptedException {
String filename = fileToUpload.getName();
int retries = 0;
while (true) {
try {
ftpClient.storeFile(filename, inputStreamFactory.getInputStream(fileToUpload));
} catch (FTPConnectionClosedException | SocketException exception) {
LOGGER.debug("Control connection lost uploading {}, continuing.", filename);
}
// This sleep is because there's an anti-malware in the servers which makes the file not to appear
// available immediately after an upload
LOGGER.debug("Waiting {} milliseconds for anti-malware protection to process file", WAIT_AFTER_UPLOAD_MILLISECONDS);
threadWrapper.sleep(WAIT_AFTER_UPLOAD_MILLISECONDS);
if (!ftpClient.isConnected()) {
connect();
}
LOGGER.debug("Checking if {} is already uploaded", filename);
if (ftpFileChecker.isFileCompleted(listFiles(null), filename, fileToUpload.length())) {
// Note this is likely to happen every time since their server will close the control
// connection quite fast and FTPClient uses it at the end of storeFile
LOGGER.debug("File {} was uploaded correctly", filename);
break;
} else {
if (++retries > MAX_RETRIES) {
throw new RemoteTimeoutException("Could not upload file, max retries exceeded");
} else {
LOGGER.info("File {} was not uploaded, retrying", filename);
}
}
}
}
public void connect() throws IOException, InterruptedException {
int retries = 0;
while (true) {
try {
ftpClient = ftpClientFactory.createFtpClient();
ftpClient.connect(server, FTP_PORT);
if (!ftpClient.login(username, password)) {
LOGGER.error("Login to FTP failed");
throw new ConfigurationException("Login to FTP failed");
}
ftpClient.enterLocalPassiveMode();
ftpClient.setControlKeepAliveTimeout(CONTROL_CONNECTION_KEEP_ALIVE_INTERVAL_SECONDS);
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
ftpClient.changeWorkingDirectory(uploadDir);
break;
} catch (FTPConnectionClosedException | SocketException exception) {
if (++retries > MAX_RETRIES) {
throw new RemoteTimeoutException("Could not upload file, max retries exceeded", exception);
} else {
LOGGER.info("Could not login, retrying");
}
}
LOGGER.debug("Sleeping {} milliseconds before trying to reconnect", WAIT_BETWEEN_CONNECT_RETRIES_MILLISECONDS);
threadWrapper.sleep(WAIT_BETWEEN_CONNECT_RETRIES_MILLISECONDS);
}
}
public FTPFile[] listFiles(String directory) throws IOException, InterruptedException {
int retries = 0;
while (true) {
try {
return ftpClient.listFiles(directory);
} catch (FTPConnectionClosedException exception) {
LOGGER.debug("Control connection lost when listing files, continuing");
} catch (SocketException exception) {
LOGGER.debug("Socket exception when listing files, continuing");
}
if (!ftpClient.isConnected()) {
connect();
}
if (++retries > MAX_RETRIES) {
throw new RemoteTimeoutException("Could not list files, max retries exceeded");
} else {
LOGGER.info("Could not list files, retrying");
}
}
}
Now I'm getting SocketException Broken Pipe on listFiles, and I can't figure it out anymore since in my local server it works perfect, but in this particular one (hint it runs on Windows NT - :( - it runs some malware protection which prevents the files to appear on the server immediately and is apparently behind some very strange firewalls, and it drops idle connections after about 5 seconds, and they won't change the configuration cause they are a big company and claim it works for everyone else).
I tried VFS and investigated over other FTP clients but none I found seem to solve the problem, even more unhelpful, most of them (like ftp4j) are not in maven central which really puts me off trying them unless there's a guarantee it will solve my problems.
Any help is welcome.
Edit: Code given reflects the starting complexity of this, my current solution is more stable with quite a lot more complexity, but it's not elegant at all, so I leave the question open in case someone cares to contribute a nice solution.
If using Spring, then consider Spring Retry. I believe the latest Maven version is:
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
<version>1.1.2.RELEASE</version>
</dependency>
I'm doing one java query with http on some search engines and here is the code of two classes:
public EventSearch(){
btsearch.addActionListener(this);
}
public void actionPerformed(ActionEvent e){
if(e.getSource()==btsearch){
try {
HttpRequest http = new HttpRequest(CatchQuery());
} catch (IOException e1) {
JOptionPane.showMessageDialog(null, "HTTP request failure.");
}
this.dispose();
}
}
public String CatchQuery(){
query=txtsearch.getText();
return query;
}
and
public class HttpRequest extends EventSearch
{
String query;
URL url;
public HttpRequest(String query) throws IOException{
// Fixed search URL; drop openConnection() at the end
try {
url = new URL("http://google.com/search?q="+query);
System.out.println(CatchQuery());
} catch (MalformedURLException e) {
JOptionPane.showMessageDialog(null, "Unable to search the requested URL");
}
// Setup connection properties (this doesn't open the connection)
URLConnection connection = url.openConnection();
connection.setRequestProperty("Accept-Charset", "UTF-8");
// Setup a reader
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
// Read line by line
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println (line);
}
// Close connection
reader.close();
}
The thing is - There are no errors regarding the code but the request is stucked. I don't receive any sort of message on my console our debug. I'm thinking of any sort of memory error since I'm working with strings but anyone has any idea of whats going wrong on?
thank you
Edit One:
public String CatchQuery(){
query=txtsearch.getText();
return query;
}
CatchQuery Simple catch the query of the txtsearch (field).
Edit Two: [Topic Solved]
Two problems:
"http://google.com/search?q="+query should be "http://google.com/search?q="+URLEncoder.encode(query), query url needs to be encoded before opening a connection, so that unsupported characters are converted to url-friendly characters
Google does not accept bot connections, you should use the Google Java API to perform searches properly
UPDATE
Google does not accept connections without the User Agent header, so you have to edit the HttpRequest class to set the user agent after creating the connection:
// Setup connection properties (this doesn't open the connection)
URLConnection connection = url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.8.1.6) Gecko/20070723 Iceweasel/2.0.0.6 (Debian-2.0.0.6-0etch1)");
connection.setRequestProperty("Accept-Charset", "UTF-8");
It works for me, test it and tell me if it works for you too.
Note: from Google ToS:
Automated queries
Google's Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries consumes resources and includes using any software (such as WebPosition Gold) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries. In addition to rank checking, other types of automated access to Google without permission are also a violation of our Webmaster Guidelines and Terms of Service.
Im trying to set a timeout on an FTP Connect using com.enterprisedt.net.ftp.FTPClient but for some reason can't get it to work.
This is my code:
public boolean testFtpDetails(String path) {
boolean success = false;
try {
FTPClient ftp = new FTPClient();
ftp.setRemoteHost("210.160.1.100");
ftp.setTimeout(3000);
ftp.connect();
ftp.login("test", "test");
success = ftp.connected();
if (success && path != null) {
ftp.chdir(path);
}
ftp.quit();
} catch (UnknownHostException e) {
} catch (IOException e) {
} catch (FTPException e) {
}
So after 3 seconds of trying to execute ftp.connect() I would expect it to timeout. However this does not happen and it takes around 15 seconds to timeout. Is there a way to change this?
Thanks,
the setTimeout method is documented as:
Set the SO_TIMEOUT in milliseconds on the underlying socket.
In this way it only affects the low level socket operations. The questions would be - what kind of timeout are you expecting to happen?
On connect ? During transfer? On idle?
If you connect to e.g. an unknown host you will also have DNS lookup taking most of the time before the timeout is even considered.
I'm working with Eclipse and the code below is the code that I use for RMI initialization.
public void init(String serviceName) throws RemoteException {
try {
String host = InetAddress.getLocalHost().getHostName();
String url = "rmi://"+ host + serviceName;
Naming.rebind(url,this);
} catch (UnknownHostException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
I'm getting an UnknownHostException.
Since I'm new to this issue, the question may be simple, but I could not handle it.
Thanks in advance.
UnknownHostException means it can't find that host at the network level. There's no handling this type of exception because it means something is broken. I'd print out the URL sent to RMI. It should look something like this:
//localhost/ServiceImTryingToAccess
If you didn't put a leading "/" on your service it might be:
//localhostServiceImTryingToAccess
And that certainly would create an UnknownHostException. You really don't need to use InetAddress.getLocalHost() as you could just simply do:
String url = "//localhost" + serviceName;
Also notice I dropped the rmi:// scheme portion of the URL. It's in the docs that's not needed.
http://docs.oracle.com/javase/1.4.2/docs/api/java/rmi/Naming.html
How do you check if you can connect to the internet via java? One way would be:
final URL url = new URL("http://www.google.com");
final URLConnection conn = url.openConnection();
... if we got here, we should have net ...
But is there something more appropriate to perform that task, especially if you need to do consecutive checks very often and a loss of internet connection is highly probable?
You should connect to the place that your actual application needs. Otherwise you're testing whether you have a connection to somewhere irrelevant (Google in this case).
In particular, if you're trying to talk to a web service, and if you're in control of the web service, it would be a good idea to have some sort of cheap "get the status" web method. That way you have a much better idea of whether your "real" call is likely to work.
In other cases, just opening a connection to a port that should be open may be enough - or sending a ping. InetAddress.isReachable may well be an appropriate API for your needs here.
The code you basically provided, plus a call to connect should be sufficient. So yeah, it could be that just Google's not available but some other site you need to contact is on but how likely is that? Also, this code should only execute when you actually fail to access your external resource (in a catch block to try and figure out what the cause of the failure was) so I'd say that if both your external resource of interest and Google are not available chances are you have a net connectivity problem.
private static boolean netIsAvailable() {
try {
final URL url = new URL("http://www.google.com");
final URLConnection conn = url.openConnection();
conn.connect();
conn.getInputStream().close();
return true;
} catch (MalformedURLException e) {
throw new RuntimeException(e);
} catch (IOException e) {
return false;
}
}
People have suggested using INetAddress.isReachable. The problem is that some sites configure their firewalls to block ICMP Ping messages. So a "ping" might fail even though the web service is accessible.
And of course, the reverse is true as well. A host may respond to a ping even though the webserver is down.
And of course, a machine may be unable to connect directly to certain (or all) web servers due to local firewall restrictions.
The fundamental problem is that "can connect to the internet" is an ill-defined question, and this kind of thing is difficult to test without:
information on the user's machine and "local" networking environment, and
information on what the app needs to access.
So generally, the simplest solution is for an app to just try to access whatever it needs to access, and fall back on human intelligence to do the diagnosis.
If you're on java 6 can use NetworkInterface to check for available network interfaces.
I.e. something like this:
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface interf = interfaces.nextElement();
if (interf.isUp() && !interf.isLoopback())
return true;
}
Haven't tried it myself, yet.
This code should do the job reliably.
Note that when using the try-with-resources statement we don't need to close the resources.
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.UnknownHostException;
public class InternetAvailabilityChecker
{
public static boolean isInternetAvailable() throws IOException
{
return isHostAvailable("google.com") || isHostAvailable("amazon.com")
|| isHostAvailable("facebook.com")|| isHostAvailable("apple.com");
}
private static boolean isHostAvailable(String hostName) throws IOException
{
try(Socket socket = new Socket())
{
int port = 80;
InetSocketAddress socketAddress = new InetSocketAddress(hostName, port);
socket.connect(socketAddress, 3000);
return true;
}
catch(UnknownHostException unknownHost)
{
return false;
}
}
}
This code:
"127.0.0.1".equals(InetAddress.getLocalHost().getHostAddress().toString());
Returns - to me - true if offline, and false, otherwise. (well, I don't know if this true to all computers).
This works much faster than the other approaches, up here.
EDIT: I found this only working, if the "flip switch" (on a laptop), or some other system-defined option, for the internet connection, is off. That's, the system itself knows not to look for any IP addresses.
InetAddress.isReachable sometime return false if internet connection exist.
An alternative method to check internet availability in java is : This function make a real ICMP ECHO ping.
public static boolean isReachableByPing(String host) {
try{
String cmd = "";
if(System.getProperty("os.name").startsWith("Windows")) {
// For Windows
cmd = "ping -n 1 " + host;
} else {
// For Linux and OSX
cmd = "ping -c 1 " + host;
}
Process myProcess = Runtime.getRuntime().exec(cmd);
myProcess.waitFor();
if(myProcess.exitValue() == 0) {
return true;
} else {
return false;
}
} catch( Exception e ) {
e.printStackTrace();
return false;
}
}
I usually break it down into three steps.
I first see if I can resolve the domain name to an IP address.
I then try to connect via TCP (port 80 and/or 443) and close gracefully.
Finally, I'll issue an HTTP request and check for a 200 response back.
If it fails at any point, I provide the appropriate error message to the user.
URL url=new URL("http://[any domain]");
URLConnection con=url.openConnection();
/*now errors WILL arise here, i hav tried myself and it always shows "connected" so we'll open an InputStream on the connection, this way we know for sure that we're connected to d internet */
/* Get input stream */
con.getInputStream();
Put the above statements in try catch blocks and if an exception in caught means that there's no internet connection established. :-)
The code using NetworkInterface to wait for the network worked for me until I switched from fixed network address to DHCP. A slight enhancement makes it work also with DHCP:
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface interf = interfaces.nextElement();
if (interf.isUp() && !interf.isLoopback()) {
List<InterfaceAddress> adrs = interf.getInterfaceAddresses();
for (Iterator<InterfaceAddress> iter = adrs.iterator(); iter.hasNext();) {
InterfaceAddress adr = iter.next();
InetAddress inadr = adr.getAddress();
if (inadr instanceof Inet4Address) return true;
}
}
}
This works for Java 7 in openSuse 13.1 for IPv4 network. The problem with the original code is that although the interface was up after resuming from suspend, an IPv4 network address was not yet assigned. After waiting for this assignment, the program can connect to servers. But I have no idea what to do in case of IPv6.
1) Figure out where your application needs to be connecting to.
2) Set up a worker process to check InetAddress.isReachable to monitor the connection to that address.
This code is contained within a jUnit test class I use to test if a connection is available. I always receive a connection, but if you check the content length it should be -1 if not known :
try {
URL url = new URL("http://www.google.com");
URLConnection connection = url.openConnection();
if(connection.getContentLength() == -1){
fail("Failed to verify connection");
}
}
catch (IOException e) {
fail("Failed to open a connection");
e.printStackTrace();
}
public boolean checkInternetConnection()
{
boolean status = false;
Socket sock = new Socket();
InetSocketAddress address = new InetSocketAddress("www.google.com", 80);
try
{
sock.connect(address, 3000);
if(sock.isConnected()) status = true;
}
catch(Exception e)
{
status = false;
}
finally
{
try
{
sock.close();
}
catch(Exception e){}
}
return status;
}
You can simply write like this
import java.net.InetAddress;
import java.net.UnknownHostException;
public class Main {
private static final String HOST = "localhost";
public static void main(String[] args) throws UnknownHostException {
boolean isConnected = !HOST.equals(InetAddress.getLocalHost().getHostAddress().toString());
if (isConnected) System.out.println("Connected");
else System.out.println("Not connected");
}
}
There are (nowadays) APIs for this, but they are platform specific:
On Android ConnectivityManager (https://developer.android.com/training/basics/network-ops/reading-network-state) does everything you need.
On Windows INetworkListManager::GetConnectivity (for which you'll need a JNI)
On generic Linux, you are probably stuck with testing if you have access to a DNS server and Google, as above.
there is probably an Apple way to do this as well
(I'd use the specific tools where available)
This have worked well for me.
try{
InetAddress addr = InetAddress.getByName("google.com" );
}catch(IOException e){
JOptionPane.showMessageDialog(new JFrame(),"No Internet connection.\nTry again later", "Network Error", JOptionPane.ERROR_MESSAGE);
}
There is also a gradle option --offline which maybe results in the behavior you want.
The following piece of code allows us to get the status of the network on our Android device
public class MainActivity extends AppCompatActivity {
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
TextView mtv=findViewById(R.id.textv);
ConnectivityManager connectivityManager=
(ConnectivityManager) this.getSystemService(Context.CONNECTIVITY_SERVICE);
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
if(((Network)connectivityManager.getActiveNetwork())!=null)
mtv.setText("true");
else
mtv.setText("fasle");
}
}
}