If I connect with Java to MySQL on my localhost server, I access instantaneously.
But if I connect outside of the localhost, from a network PC (192.168.1.100), it is very slow (4-5 seconds).
And, if I connect from a public IP to my MY SQL server, it is also very slow (6 seconds or more).
The "why" is already been answered. It's just the network latency.
You're probably also interested in how to "fix" it. The answer is: use a connection pool. If you're running a Java webapplication, use the webserver-provided connection pooling facilities. To take Tomcat as an example, check this manual. If you're running a Java desktop application, use a decent connection pool implementation like c3p0 (tutorial here).
Because your computer needs time to send packets to external servers and they need time to send packets back. It's called network latency, and is not an issue with Java specifically, but a general network issue.
It will always take longer to make a connection across the network than to make the same connection locally. However, assuming you have a fairly typical local network, 4-5 seconds sounds a bit extreme. My guess (and it is just a guess) would be that the majority of the extra time is being consumed by network name resolution (i.e. DNS and/or netbios).
I would suggest that you try the connection using a numeric IP address, rather than a name.
Network latency plus connection creation time would be my guess. I don't know what else you have between the client machine and the MySQL server.
4 seconds on connecting could be a DNS problem and cannot be just a pure network latency.
Try starting MySQL server with "skip-name-resolve" parameter to skip resolving client's IP into hostname. Prior to that, make sure your grant tables are based on IPs and 'localhost' instead of symbolic names.
Related
I would like to connect to a database that runs locally on one of our network computers here at work. I can connect to it just fine with the application that I developed that uses Java's sql driver manager. Now I would like to distribute it to the different computers on the network. To do this, I need a url that will point to the database through the network. The database is listening to port 1434. (Static URL string is "//localhost:1434")
MatsysUI.setConnection(DriverManager.getConnection("jdbc:sqlserver:" + MatsysIO.getStaticURL(), txtUser.getText(), txtPassword.getText()));
Problem is, I don't know where to start to find that, and I would like to avoid using an internet connection to connect to this database. Is there a way to route the connection to the network computer, then to its local port?
There are several possible answers to this.
I just want to restate your situation to make sure I understand:
You have an application, written in Java, which requires access to a SQL Server database.
You want to distribute multiple copies of that application to different client machines.
The client machines and the database server are all on a local network.
SQL Server is listening on port 1434
You need to construct the JDBC connection string on the client machines to access SQL Server.
You give the current connection string as //localhost:1434; in that scheme, localhost is the hostname. You can replace this with the fully qualified domain name of the SQL Server machine (this is almost certainly the simplest option). Using the FQDN allows you to replace the machine, or load balance it, etc. without worrying about the connections - but if the machine name ever changes, all your connections break!
You can also connect by IP address - this allows you to change the machine name, but obviously means the IP address can never change. In most circumstances, that's a bad thing.
Your final option is to look for alL SQL Servers on your network, and show them in a drop down for your user (your code suggests you're using username and password inputs). This means you don't have to distribute a new properties file if you want to change your server details.
The system I am developing potentially has a very large number of clients (lets say one million) that need to periodically update a central server with some information. Clients are written in Java.
The specific use-case is that the server backend needs to have an up to date mapping of IP address to clients. But the client IPs are dynamic and subject to (effectively random) change.
The solution I have in mind requires the clients to ping the server to update their IP. The period ideally should be once every minute, but even 1 ping/10 mins is acceptable.
My questions, in sequence:
1M pings per 1 min is over 10k/sec. So first off I want to know
the approaches can scale to handle such a load. This is to know the options available.
Assuming you have more than one solution in mind, which of these
would be the most economical? The cost effectiveness is critically important. I don't have my own data center or
static and fat end-point on the net, so the server application will
need to run on some sort of provider or ultimately on the cloud.
Notes:
I considered running the server from home using my own ISP provided connection, but I am neither sure of the performance issues, nor what my ISP will think about a constant stream of pings.
I can't see how the server can auto-discover these IP changes.
Erik, your problem is much simpler than it seems to have been made to sound.
This problem been around for a decade maybe two. No need to re-invent the wheel here.
Why Polling/Pinging is a Bad Idea
The dynamic IPs provided by ISPs can have a variable lease time, but will often be at least 24-72 hours. Pinging your server every 1-10m will be a horrible waist of resources potentially making over a 4,320 useless HTTP requests PER CLIENT in a 72 hour period. Each request will be say around 300 bytes * 4,320 wasted http requests equals 1.3mb wasted bandwidth multiplied by your target client count of 1 million clients, you are talking about a monthly wasted bandwidth of ~1.2 TB! And that's just the wasted bandwidth, not the other bandwidth you might need to run your app and provide useful info.
The clients need to be smarter than just pinging frequently. Rather they should be able to check if their IP address matches the DNS on startup, then only when the IP changes, send a notification to the server. This will cut down your bandwidth and server processing requirements by thousands of times.
What you are describing is Dynamic DNS
What you are talking about is "Dynamic DNS" (both a descriptive name for the technology and also the name of one company that provides a SaaS solution).
Dynamic DNS is quite simply a DNS server that allows you to very rapidly change the mapping between a name and an IP address. Normally this is useful for devices using an ISP which only provides dynamic IPs. Whenever the IP changes for the router/server on a dynamic IP it will inform the Dynamic DNS server of the change.
The defacto standard protocol for dynamic DNS is well documented. Start here: DNS Update API, I think the specifics you are looking for are here: DynDNS Perform Update. Most commercial implementations out there are very close to the same protocol due to the fact that router hardware usually has a built in DynDNS client which everyone wants to use.
Most routers (even cheap ones) already have Dynamic DNS clients built into them. (You can write your own soft client, but the router is likely the most efficient location for this as your clients are likely being NAT'd with a private IP - you can still do it but at a cost of more bandwidth for public IP discovery)
A quick google search for "dynamic DNS java client" brings up full source projects like this one: Java DynDNS client (untested, just illustrating the power of search)
Other Considerations for your System Architecture
Lets say the IP-client mapping thing gets resolved. You figured it all out and it works perfectly, you always knows the IP for each client. Would you then have a nice reliable system for transferring files to clients from mobile devices? I would say no.
Both mobiles and home computers can have multiple connection types, Wi-Fi, Cellular Data, maybe wired data. Each of these networks may have different security systems in place. So a connection from a cellular data mobile to a wifi laptop behind a home router is going to look very different than a wifi mobile device connecting to laptop on the same wifi network.
You may have physical router firewalls to contend with. Also home computers may have windows firewall enabled, maybe norton internet security, maybe symantec, maybe AVG, maybe zone alarm, etc... Do you know the firewall considerations for all these potential clients?
Maybe you could use SIP as protocol for that purpose ?
Probably the java SIP libs already solved your problem.
Nice app by the way.
I would suggest better tweak you java program to know the IP change and then only hit the web service.
You can do it like,
on your java program initiation extract the IP of machine and store
it in Global variable or better some property file.
Run a batch process/scheduler which will check your IP every 30sec/1 minute for change.Java Quartz Scheduler will come very handy for you.
Invoke the web service in case of a change of IP.
This way it reduces your server role and thus traffic and connections.
You could create your own protocol on top of UDP, for example XML based. Define 3 messages:
request - client requests a challenge from server
challenge - server replies with challenge (basically a random number)
response - client sends username and hashed password + challenge back to the server
It's lightweight and not too traffic-heavy. You can load-balance it to multiple servers at any layer or using load-balancer.
Any average PC could handle million such hits per minute, provided you do server-side in C/C++ (I don't know about java network performance)
Please have a look at how no-ip works. Your requirement is exactly same as what it does.
Do I have the use case right? A community of users all want to receive pictures from each other? You don't want to host the images on the server but broadcast them directly to all the users?
There are two questions here. The first question is "how to know if my own WAN IP address has changed."
If you are not NATed then:
InetAddress.getLocalHost()
will tell you your IP address.
If you are NATed, then using dynamic DNS and resolving your own host name will work.
The second question is something like "How to share pictures between hosts which come and go on the internet".
The possible solution space includes:
IP Multicast, probably with Forward Error Correction and Carouseling, e.g. FLUTE.
File Swarming - e.g. bittorrent.
A Publish/Subscribe message bus solution using Jabber, AMQP, JMS, STOMP or similar. Suitable implementations include RabbitMQ, ActiveMQ, etc. JMS Topics are a key concept here.
The solution should avoid the massive overheads of doing things at the IP level.
I have a very short Java application that just opens a connection to a remote MySQL database, reads some data, prints it, and exits. The most time-consuming part of the application is the database connection.
Currently I have only a single thread, and my only concern is to save the time of opening the connection.
I thought of several ways to make it faster, but it turned out they do not help:
Connection Pooling - doesn't help because the pool lives only only during a single run of the application. When the application is terminated, the pool is gone, and when I re-run the application, I have to re-open all the connections in the pool.
mysql-proxy - connects only to the local server: mysql-proxy for a remote MySQL server
TCP/IP server - I thought of holding a local TCP/IP server that will keep a persistent open connection and send it to a TCP/IP client on request. However, Connection objects cannot be serialized, so I have no way to pass the Connection object from client to server.
Any other option?
Generally connection to a DB is a most time-consuming operation. If the application is to be started and stopped then there is little that you can do.
Using connection-pooling in a web-server and call that by running your app which talks to the web server using JSON might be an option.
You said you have a very short application so your 3rd option might work if you put the database logic into you "option 3 TCP/IP server" and just forward the results to your connecting client. This is a typical application server pattern.
Another thing you should consider about network look up https://stackoverflow.com/q/3641155/1055715 which Marc B has mentioned in his comment.
It turns out the best solution is to use mysql-proxy with a script that handles connection pooling (a combination of my first two options). I found one such script here:
http://forge.mysql.com/tools/tool.php?id=151
It was probably written for an older version of mysql-proxy, so I had to fix it (if anyone need the fixed version - write me).
It works like a charm - I run the exact same application as before, the only change is in the connection string: instead of connecting to "qa-srv:3308" (the remote server) I connect to "127.0.0.1:4040" (the proxy server).
We have a Java application running on a portable device and the JVM we're using (CReMe) appears to have a bug where it caches negative DNS lookups, even when we tell it not to. Thus, when we try to make a connection to our server and the DNS lookup fails because the network connection isn't established yet, the application is stuck because it will never perform the lookup again, even when connectivity is restored.
We've tried testing by opening a Socket to a hard-coded IP address, but obviously there's no guarantee that a given IP address won't change in the future. Can anyone suggest another way to verify that we have network connectivity?
You could have a list of IPs and try to open a connection. They could be IPs of machines on your LAN or they could be Google or Facebook's IP.
If you can't connect to any IP on the list, you can assume the network is down.
When the network is up, you can update the list with a DNS lookup.
You could try custom DNS client like dnsjava.
Maybe you can try to use the DNS servers of Google? Doubt they will ever change, and pretty sure they are usually up :)
8.8.8.8
8.8.4.4
I have a problem with MantaRay JMS: I use a static world map because autodiscovery wouldn't work in our network. If more than 10 peers are offline, I get an error 4226.
The problem is: Microsoft set a limit of 10 half-open connections with Windows XP SP2. MantaRay tries to contact every peer, and starts a lot of connections. The first 10 connections are Ok, then when the 11th starts, our software must wait for another connection to time out. Any other program trying to access the network on the same PC times out.
Strange thing is: on some PC the connection times out after 1-2 seconds, and the problem has almost no consequences, on some other, we have to wait 10 or 20 seconds. But according to Microsoft, there's no way to configure the default TCP connect timeout directly, and there are other factors (network switches, routers, VPN...) which could influence that.
I looked at the MantaRay source code, and tried to find a way to set the TCP connect timeout, but MantaRay uses SohetChannels instead of "regular" sockets, and the connect() method has no timeout. Am I missing something?
You could also patch the TCP/IP connection limit of WinXP... if you don't mind using such things. There are several sites offering patches. Just search Google for "change winxp tcp connection limit" and you'll find most of them. But use those tools on your own risk. Patching the code to work around that limit should be a better approach.
Problem solved.
I replaced the whole MantaRay with a much simpler JMS provider I wrote: I send a first test message over UDP, a peer is allowed to open a TCP connection only after this first message was received.
This taught me to be careful when using open-source (GPL) software.