I'm trying to write a small java program that connects to a twitter search URL (which returns a JSON list of tweets) using the URL connection libary.
My code which is taken from the java tutorials looks like :
public static void main(String[] args) throws Exception {
URL oracle = new URL("http://search.twitter.com/search.json?q=hi");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
but for some reason I keep getting the following Exception:
in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
I don't know if this is something due to the way I've written the code, and eclipse setting or something to do with my network. I do have a proxy server configured for internet access. As far as I know this is properly configured because I am getting updates and can install new software through eclipse. Do I need to put the proxy information in the URL method somehow or is something else the problem.
URL rely on System properties for proxy, try setting proxy like this:
System.setProperty("http.proxyHost", "yourproxyserver");
System.setProperty("http.proxyPort", "portnumber");
Unfortunately, a correct proxy setup in Eclipse seems not to help with proxying Java programs started in Eclipse. Similarly, setting the Java system settings to use the systemwide proxy settings doesn't either. Not when you have a proxy that requires authentication, anyway.
As Thomas Johan Eggum said, if you have a "normal," non-authenticating proxy then setting the two JVM variables http.proxyHost and http.proxyPort either in the command line via -D or programmatically (see below) will take care of things.
For an authenticating proxy server, i.e. one that wants to see a user ID and password, many people recommend setting http.proxyUser and http.proxyPassword. That's bad advice, because these don't work. Apparently they are not defined in the Java docs.
Unfortunately, it looks like the way to "do" authentication is to use an Authenticator, programmatically. If you're going to do that, you might as well do the whole thing programmatically, i.e. including host and port. Here's how I got that to work:
public static void main(String[] args) {
try {
System.setProperty("http.proxyHost", "my.proxy.host");
System.setProperty("http.proxyPort", "8080-or-whatever-proxy-port");
Authenticator.setDefault(new DummyAuthenticator());
/* do your main program stuff */
} catch (Exception e) {
e.printStackTrace();
}
}
private static class DummyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(
"my-user-id", "my-password".toCharArray()
);
}
}
Related
I've developed a decent amount of multiplayer games lately in java (board/turn based games running on a tcp connection with data in/out streams) but they only work locally, I want to go a step further and make them run online, what do I need to change to make it work?
It seems like there are many ways to accomplish that yet i don't know where to start, as I don't know much about networking.
Besides that I don't have a real server, I'm only using my home router and my pc.
So here is what I've tried so far:
I enabled port forwarding in my router and I think it works (I used a port forwarding checking tool online)
I created a dynamic DNS for my public ip using noip.com
So the server side should be fine at least, my problem is with the client side, the client's socket won't connect to my public ip, when I searched for a solution I concluded that the client shouldn't be in the same LAN where the server is, so I used a mobile hotspot as a second network and tried again, but got the same results. (connection refused exception)
is it because of the mobile hotspot (should I use another router) ?
or is it just some coding tweaks ?
This is a minimal server class example
public static void main(String[] args) {
try {
ServerSocket ss = new ServerSocket(50895);
Socket cs = ss.accept();
DataOutputStream out = new DataOutputStream(cs.getOutputStream());
DataInputStream in = new DataInputStream(cs.getInputStream());
out.writeInt(123456);
} catch (IOException e) {
e.printStackTrace();
}
}
And this a minimal client class example
public static void main(String[] args) {
try {
String ip = Inet4Address.getByName("my Dynamic IP address").getCanonicalHostName();
System.out.println(ip);
InetSocketAddress sa = new InetSocketAddress(ip, 50895);
Socket cs = new Socket();
cs.connect(sa);
DataOutputStream out = new DataOutputStream(cs.getOutputStream());
DataInputStream in = new DataInputStream(cs.getInputStream());
System.out.println("received : " + in.readInt());
} catch (IOException e) {
e.printStackTrace();
}
}
The method that i've tried always gives me a connection refused exception, so any solution would be appreciated.
I found a software solution to the problem through virtual lan (using LogMeIn Hamachi) which is basicly used for creating game servers, you just need to create an account and a network so your friends can join it, after that everything should be fine and runs as if it's locally hosted.
You can download the software through here but there are many alternatives to choose from, that one gives you up to 5 people on the network for free, it's not the best, but still a free solution.
I still want more of a java solution to this problem that doesn't require a third-party software.
I am trying to create a basic connection and login to an Openfire server that I have installed. I have the following user in my users database which I created through the Openfire web admin interface:
User: user
Password: 12345678
I can connect fine to the server as the connection returns true in my sout. The problem is when it tries to log in I get the following error:
org.jivesoftware.smack.sasl.SASLErrorException: SASLError using DIGEST-MD5: not-authorized
I have the following code:
private XMPPConnection connection;
public void connect(String serverIP) {
try {
System.setProperty("smack.debugEnabled", "true");
ConnectionConfiguration config = new ConnectionConfiguration(serverIP, 5223);
config.setDebuggerEnabled(true);
config.setSocketFactory(new DummySSLSocketFactory());
config.setSecurityMode(ConnectionConfiguration.SecurityMode.enabled);
config.setCompressionEnabled(true);
connection = new XMPPTCPConnection(config);
connection.connect();
System.out.println("Connected: " + connection.isConnected());
connection.login("user", "12345678");
System.out.println("Logged in: " + connection.isAuthenticated());
} catch (SmackException | IOException | XMPPException ex) {
ex.printStackTrace();
}
}
public static void main(String[] args) {
connectionHandler test = new connectionHandler();
test.connect("localhost");
}
If anyone can correct what I am doing wrong I would be really grateful.
I have also tried the username as the email would be for example
user#localhost.com
or
user#localhost
I finally managed to find the answer to this. The problem (possibly not even a problem) was that the authentication methods weren't set in the server config and as default allowed all methods. The first one chosen in java seems to be DIGEST-MD5 which was what was causing the errors. To fix this I added:
<sasl>
<mechs> PLAIN </mechs>
</sasl>
before the last closing tag of the openfire.xml found in the config folder of the server. This can also be changed in the ofproperty database table for the column called sasl.mechs.
Hopefully this helps someone (possibly me) in the future.
P.S. this is unsecure if not using SSL (port 5223 by default)
SASLError using DIGEST-MD5: not-authorized
This is most likely caused because you did not configure the correct XMPP domain (/service name) in your ConnectionConfiguration. DIGEST-MD5 would not only fail if the username or password is wrong, but also if the wrong XMPP domain is used.
I have set up a local proxy server for request logging but my java code ignores it and connects directly (Windows XP, JDK 1.7). Web browsers work with it. So I wrote test code for discussion that seems to connect directly even if a (bogus) proxy is specified. With the bogus proxy, I would expect connection failure but the code succeeds, connecting directly:
System.setProperty("http.proxyHost", "localhost");
System.setProperty("http.proxyPort", "12345");
System.setProperty("http.nonProxyHosts", "noNonProxyHost.com");
URL url = new URL("http://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html");
InputStream in = url.openStream();
System.out.println("Connection via bogus proxy succeeded");
The code is run as standalone Java, no Maven, no applet, no container. I have a direct internet connection.
In your case using java.net.URL(), if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
This sample code below happily fetches the URL and displays it, without error, even when run with bogus proxy settings. -Dhttp.proxyHost=bogus -Dhttp.proxyPort=2345 but will talk to my local proxy localhost port 8888 if set correctly
import java.io.*;
import java.net.URL;
import java.util.*;
public class URLClient {
private static String sUrl = "http://www.apache.org/";
public static void main(String[] args) {
try {
URL url = new URL(sUrl);
InputStream is = url.openStream();
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
String output = s.hasNext() ? s.next() : "";
System.out.println(output);
} catch(Throwable e) {
System.err.println("exception");
}
}
}
The problem I was originally having with http.proxyHost and http.proxyPort being ignored (Google led me to your question) was that those settings are completely ignored by apache.commons.httpClient because it uses its own sockets, as described here.
http://cephas.net/blog/2007/11/14/java-commons-http-client-and-http-proxies/
I have faced a similar problem recently. First of all, one part of the above answer from Daemon42 explains pretty well, why the bogus proxy server didn't lead to a failure of the program:
if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
Still, your actual question was, why the proxy server configured via the operating system is not used by the Java application. As stated in the Oracle documentation (https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html), the system proxy settings are not evaluated by Java by default. To do so, you have to set the value of the system property "java.net.useSystemProxies" to the value "true".
You can set that system property on the command line, or you can edit the JRE installation file jre/lib/net.properties, that way you have to change it only once on a given system.
Hi I am trying to write a java client for secure hbase.
I want to do kinit also from code itself for that i`m using the usergroup information class.
Can anyone point out where am I going wrong here?
this is the main method that Im trying to connect o hbase from.
I have to add the configuration in the CONfiguration object rather than using the xml, because the client can be located anywhere.
Please see the code below:
public static void main(String [] args) {
try {
System.setProperty(CommonConstants.KRB_REALM, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF, "krb.realm"));
System.setProperty(CommonConstants.KRB_KDC, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF,"krb.kdc"));
System.setProperty(CommonConstants.KRB_DEBUG, "true");
final Configuration config = HBaseConfiguration.create();
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, AUTH_KRB);
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, AUTHORIZATION);
config.set(CommonConfigurationKeysPublic.FS_AUTOMATIC_CLOSE_KEY, AUTO_CLOSE);
config.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, defaultFS);
config.set("hbase.zookeeper.quorum", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.host"));
config.set("hbase.zookeeper.property.clientPort", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.port"));
config.set("hbase.client.retries.number", Integer.toString(0));
config.set("zookeeper.session.timeout", Integer.toString(6000));
config.set("zookeeper.recovery.retry", Integer.toString(0));
config.set("hbase.master", "gauravt-namenode.pbi.global.pvt:60000");
config.set("zookeeper.znode.parent", "/hbase-secure");
config.set("hbase.rpc.engine", "org.apache.hadoop.hbase.ipc.SecureRpcEngine");
config.set("hbase.security.authentication", AUTH_KRB);
config.set("hbase.security.authorization", AUTHORIZATION);
config.set("hbase.master.kerberos.principal", "hbase/gauravt-namenode.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.master.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
config.set("hbase.regionserver.kerberos.principal", "hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.regionserver.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setConfiguration(config);
UserGroupInformation userGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setLoginUser(userGroupInformation);
User user = User.create(userGroupInformation);
user.runAs(new PrivilegedExceptionAction<Object>() {
#Override
public Object run() throws Exception {
HBaseAdmin admins = new HBaseAdmin(config);
if(admins.isTableAvailable("ambarismoketest")) {
System.out.println("Table is available");
};
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable("ambarismoketest");
admins.close();
System.out.println(table.get(new Get(null)));
return table.get(new Get(null));
}
});
System.out.println(UserGroupInformation.getLoginUser().getUserName());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I`m getting the following exception:
Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:110)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:146)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:762)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:354)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:883)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:880)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:880)
... 33 more
Any pointers would be helpful.
The above works nicely, but I've seen a lot of folks struggle with setting all of the right properties in the Configuration object. There's no de-facto list that I've found of exactly what you need and don't need and it is painfully dependent on your cluster configuration.
The surefire way is to have a copy of your HBase configurations in your classpath, since your client can be anywhere as you mentioned. Then you can add the resources to your object without having to specify all properties.
Configuration conf = HBaseConfiguration.create();
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Here were some sources to back this approach:
IBM,
Scalding (Scala)
Also note that this approach doesn't limit you to actually use the internal Zookeeper principal and keytab, i.e. you can create keytabs for applications or Active Directory users and leave the internally generated keytabs for the daemons to authenticate amongst themselves.
Not sure if you still need help. I think setting the "hadoop.security.authentication" property is missing from your snippet.
I am using following code snippet to connect to secure HBase (on CDH5). You can give a try.
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
in Jdk 1.8, you need set
"System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");"
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
quote:
http://hbase.apache.org/book.html#trouble.client
question: 142.9
I think the best is https://scalding.io/2015/02/making-your-hbase-client-work-in-a-kerberized-environment/
To make the code work you don’t have to change any line from the one written in the top of this post, you just have to make your client able to access the full HBase configuration. This just implies to change your running classpath to:
/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hbase/conf:target/scala-2.11/hbase-assembly-1.0.jar
This will make everything run smoothly. It is specific for CDH 5.3 but you can adapt it for your cluster configuration.
PS No need in this:
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Because HBaseConfiguration has
public static Configuration addHbaseResources(Configuration conf) {
conf.addResource("hbase-default.xml");
conf.addResource("hbase-site.xml");
I have a very simple code that uses HttpURLConnection to access some web site via proxy
System.setProperty("java.net.useSystemProxies", "true");
System.out.println("Proxy: " + ProxySelector.getDefault().select(new URI(urlS)));
URL url = new URL(urlS);
HttpURLConnection ic = (HttpURLConnection)url.openConnection();
ic.connect();
For some reason, Java thinks that I need SOCKS proxy, not http, throwing the following exception:
ERROR: Can't connect to SOCKS proxy:Connection timed out: connect
If you are having this issues on Windows, you may run into a Java bug.
Java treats any system proxy setting as SOCKS. You have to either disable useSystemProxies or don't use proxy in Windows.
If proxy is needed, try to uncheck "Use the same proxy server for all protocols", making sure the field for the SOCKS proxy is blank. That fixed our problem.
The real problem is that Java assumes that the "Use the same proxy server for all protocols" check affects SOCKS proxy too (I don't know the logic behind this dialog in Windows, but it is, at least, confusing)
If the check is set, you get proxies enabled for both HTTP and SOCKS, wich is very unlikely to be the desired configuration.
One way to solve it is unchecking the check and leaving blank the SOCKS field.
I finally solved it creating a ProxySelector wich first calls the default selector and if it finds the same configuration for HTTP and SOCKS connections, it omits the SOCKS proxy.
public class SocksFixerProxySelector extends ProxySelector {
ProxySelector base;
public SocksFixerProxySelector() {
base = ProxySelector.getDefault();
}
#Override
public List<Proxy> select(URI uri) {
List<Proxy> baseList = base.select(uri);
try {
if (uri.getScheme().equals("socket")) {
Proxy socksProxy = findByType(baseList, Type.SOCKS);
if (socksProxy != null) {
URI httpTestUri = new URI("http", uri.getHost(), uri.getPath(), uri.getFragment());
Proxy httpProxy = findByType(base.select(httpTestUri), Type.HTTP);
if (httpProxy != null && socksProxy.address().equals(httpProxy.address())) {
// Quitamos SOCKS
List<Proxy> filteredList = new ArrayList<>(baseList);
filteredList.remove(socksProxy);
return filteredList;
}
}
}
} catch (Exception e) {
}
return baseList;
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
base.connectFailed(uri, sa, ioe);
}
private Proxy findByType(List<Proxy> proxies, Proxy.Type type) {
for (Proxy proxy : proxies) {
if (proxy.type() == type)
return proxy;
}
return null;
}
Maybe a better solution would be to inspect the registry and detect the right settings, but I didn't want to mess with Windows specific code (and all those script settings looked bad, too )
You need to use the http.proxyHost system property instead. See http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html for details.
java -Dhttp.proxyHost=webcache.mydomain.com GetURL
Check that something has not set the "socksProxyHost" property in the Systems properties.
EDIT
The "useSystemProxies" property is described thus:
"On recent Windows systems and on Gnome 2.x platforms it is possible to tell the default ProxySelector to use the system proxy settings (both recent versions of Windows and Gnome 2.x let you set proxies globally through their user interface). If the system property java.net.useSystemProxies is set to true (by default it is set to false for compatibility sake), then the default ProxySelector will try to use these settings."
So, assuming that you have not supplied your own ProxySelector class, you should also check the system proxy settings to ensure that they don't say to use SOCKS.