We Keep Coding

Sending HTTPS GET request from android to python is not working

I'm trying to establish a https connection between an android app (client) an my laptop (server).
My https server is running as python script (with a letsencrypt certificate) which works fine as long as I try to connect it with chrome or another python script.
Now I wanted to implement the client in my android app. Therefore I added the permission to the AndroidManifest.xml:
<uses-permission android:name="android.permission.INTERNET"/>
and added following lines to my MainActivity.java (based on HttpURLConnection Reference on Android Developers!:
public void onButtonClicked(String message) {
try {
URL url = new URL("https://foo.bar.com/");
HttpsURLConnection urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
OutputStream output = new BufferedOutputStream(urlConnection.getOutputStream());
} catch (IOException e) {
e.printStackTrace();
}
mSecondFragment.updateMessage(message);
}
At the moment I just want to establish a connection to my https server and send a simple GET request without receiving any data. But my goal would be to parse an additional key value pair together with the GET request that will be processed by the server:
"https://foo.bar.com?a=1"
I tried to keep it as simple as possible (that's the reason why I wanted to use java.net.HttpURLConnection) but I assume the problem is not as trivial as I expected.
Maybe someone ran into the same problem and can help my with that :)
EDIT (Thanks to #atomicrat2552 and #petey):
I added an additional class that handles the request as an AsyncTask:
public class NetworkConnection extends AsyncTask<String, Void, NetworkConnection.Result> {
static class Result {
public String mResultValue;
public Exception mException;
public Result(String resultValue) {
mResultValue = resultValue;
}
public Result(Exception exception){
mException = exception;
}
}
protected NetworkConnection.Result doInBackground(String... urls) {
Result result = null;
HttpsURLConnection urlConnection = null;
try {
URL url = new URL(urls[0]);
urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
//urlConnection.connect();
result = new Result("Done");
}catch(Exception e) {
result = new Result(e);
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
return result;
}
}
This leads to a simplified onButtonClick method in MainActivity.java:
NetworkConnection nwConn = new NetworkConnection();
public void onButtonClicked(String message) {
nwConn.execute("https://foo.bar.com");
mSecondFragment.updateMessage(message);
}
Again I tried to simplify the code in order to get a small working code a can extend later.
The app doesn't crash anymore but my server still doesn't show any requests. If I'm using the browser on my phone everything works just fine. Any idea?

The most common pitfall I see here is that network code cannot run on the UI thread, so you must use some sort of background worker to do your network calls. The developer site has a basic guide on how to do this.

How to get a proxy-less connection in Java?

How do I avoid going through the ProxySelector when making a connection with URLConnection or rather how to get a connection guaranteed free of whatever proxies Java knows about ?
I thought this was what Proxy.NO_PROXY was for. Quoting from the Javadoc:
A proxy setting that represents a DIRECT connection, basically telling
the protocol handler not to use any proxying
Yet such a connection will still go through the ProxySelector. I don't get it ??
I've made a small test to prove my point:
public static void main(String[] args) throws MalformedURLException, IOException {
ProxySelector.setDefault(new MyProxySelector());
URL url = new URL("http://foobar.com/x1/x2");
URLConnection connection = url.openConnection(Proxy.NO_PROXY);
connection.connect();
}
and a dummy ProxySelector which does nothing but log what is going on:
public class MyProxySelector extends ProxySelector {
#Override
public List<Proxy> select(URI uri) {
System.out.println("MyProxySelector called with URI = " + uri);
return Collections.singletonList(Proxy.NO_PROXY);
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {}
}
which prints:
"MyProxySelector called with URI = socket://foobar.com:80"
(Note how the protocol has changed from http to socket)
I can of course create my ProxySelector in such a way so that it always returns Proxy.NO_PROXY if the URI scheme is socket but I guess there would be occasions where there's a SOCKS proxy on the site and then it wouldn't be true.
Let me restate the question: I need a way to make sure a specific URLConnection doesn't use a proxy, regardless of what System Properties may be set or what ProxySelector is installed.

This is tracked as JDK bug 8144008.

Java HTTP Request Fail

I'm doing one java query with http on some search engines and here is the code of two classes:
public EventSearch(){
btsearch.addActionListener(this);
}
public void actionPerformed(ActionEvent e){
if(e.getSource()==btsearch){
try {
HttpRequest http = new HttpRequest(CatchQuery());
} catch (IOException e1) {
JOptionPane.showMessageDialog(null, "HTTP request failure.");
}
this.dispose();
}
}
public String CatchQuery(){
query=txtsearch.getText();
return query;
}
and
public class HttpRequest extends EventSearch
{
String query;
URL url;
public HttpRequest(String query) throws IOException{
// Fixed search URL; drop openConnection() at the end
try {
url = new URL("http://google.com/search?q="+query);
System.out.println(CatchQuery());
} catch (MalformedURLException e) {
JOptionPane.showMessageDialog(null, "Unable to search the requested URL");
}
// Setup connection properties (this doesn't open the connection)
URLConnection connection = url.openConnection();
connection.setRequestProperty("Accept-Charset", "UTF-8");
// Setup a reader
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
// Read line by line
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println (line);
}
// Close connection
reader.close();
}
The thing is - There are no errors regarding the code but the request is stucked. I don't receive any sort of message on my console our debug. I'm thinking of any sort of memory error since I'm working with strings but anyone has any idea of whats going wrong on?
thank you
Edit One:
public String CatchQuery(){
query=txtsearch.getText();
return query;
}
CatchQuery Simple catch the query of the txtsearch (field).
Edit Two: [Topic Solved]

Two problems:
"http://google.com/search?q="+query should be "http://google.com/search?q="+URLEncoder.encode(query), query url needs to be encoded before opening a connection, so that unsupported characters are converted to url-friendly characters
Google does not accept bot connections, you should use the Google Java API to perform searches properly
UPDATE
Google does not accept connections without the User Agent header, so you have to edit the HttpRequest class to set the user agent after creating the connection:
// Setup connection properties (this doesn't open the connection)
URLConnection connection = url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.8.1.6) Gecko/20070723 Iceweasel/2.0.0.6 (Debian-2.0.0.6-0etch1)");
connection.setRequestProperty("Accept-Charset", "UTF-8");
It works for me, test it and tell me if it works for you too.
Note: from Google ToS:
Automated queries
Google's Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries consumes resources and includes using any software (such as WebPosition Gold) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries. In addition to rank checking, other types of automated access to Google without permission are also a violation of our Webmaster Guidelines and Terms of Service.

JUnit test on URLConnection, use EasyMock?

Hey, have been trying to work this out for last day or so but hitting brick wall. Trying to unit test this bit of code. But not sure if need to use EasyMock or not?? Seem few examples online but seem to be using older techniques.
public boolean verifyConnection(final String url) {
boolean result;
final int timeout = getConnectionTimeout();
if (timeout < 0) {
log.info("No need to verify connection to client. Supplied timeout = {}", timeout);
result = true;
} else {
try {
log.debug("URL: {} Timeout: {} ", url, timeout);
final URL targetUrl = new URL(url);
final HttpURLConnection connection = (HttpURLConnection) targetUrl.openConnection();
connection.setConnectTimeout(timeout);
connection.connect();
result = true;
} catch (ConnectException e) {
log.warn("Could not connect to client supplied url: " + url, e);
result = false;
} catch (MalformedURLException e) {
log.error("Malformed client supplied url: " + url, e);
result = false;
} catch (IOException e) {
log.warn("Could not connect to client supplied url: " + url, e);
result = false;
}
}
return result;
}
It just take's in a url checks its valid and returns T or F.

I have always observed that Mocking Can be avoided as much as possible because it can lead to difficult to maintain JUnit tests and defeat the whole purpose.
My suggestion would be to create a temporary server on your local machine from a JUnit itself.
At the beginning of JUnit you can create a server(not more than 10-15 lines of coding required) using Java sockets and then in your code pass the URL for the local server. This way you are reducing mocking and ensuring maximum code coverage.
Something like this -
public class SimpleServer extends Thread {
public void run() {
try {
serverSocket = new ServerSocket(port);
while (true) {
Socket s = serverSocket.accept();
}
}
catch (IOException e) {
e.printStackTrace();
}
finally {
serverSocket = null;
}
}
}

If you want to mock this method, I'd recommend passing in the URL rather than the String. Don't have your method create the URL it needs; let the client create the URL for you and pass it in. That way your test can substitute a mock if it needs to.
It's almost a dependency injection idea - your method should be given its dependencies and not create them on its own. The call to "new" is the dead giveaway.
It's not a drastic change. You could overload the method and have two signatures: one that accepts a URL string and another that accepts the URL itself. Have the first method create the URL and call the second. That way you can test it and still have the method with the String signature in your API for convenience.

Trying to set up mock implementation of the HttpURLConnection. Like
public class MockHttpURLConnection extends HttpURLConnection {'
then added method to class to override
' protected HttpURLConnection createHttpURLConnection(URL url)
throws IOException {
return (HttpURLConnection) url.openConnection();
}
So test looking something like this:
#Test
public void testGetContentOk() throws Exception
{
String url = "http://localhost";
MockHttpURLConnection mockConnection = new MockHttpURLConnection();
TestableWebClient client = new TestableWebClient();
client.setHttpURLConnection(mockConnection);
boolean result = client.verify(url);
assertEquals(true, result);
}
#Test
public void testDoesNotGetContentOk() throws Exception
{
String url = "http://1.2.3.4";
MockHttpURLConnection mockConnection = new MockHttpURLConnection();
TestableWebClient client = new TestableWebClient();
client.setHttpURLConnection(mockConnection);
boolean result = client.verify(url);
assertEquals(false, result);
}
/**
* An inner, private class that extends WebClient and allows us
* to override the createHttpURLConnection method.
*/
private class TestableWebClient extends WebClient1 {
private HttpURLConnection connection;
/**
* Setter method for the HttpURLConnection.
*
* #param connection
*/
public void setHttpURLConnection(HttpURLConnection connection)
{
this.connection = connection;
}
/**
* A method that we overwrite to create the URL connection.
*/
#Override
public HttpURLConnection createHttpURLConnection(URL url) throws IOException
{
return this.connection;
}
}
First part passed but is getting true for false dummy test, thanks for feedback back so far best site I have found for help. So let me know if think on right track

"Could not load known_hosts" exception using SSHJ

I am getting an exception while using SSHJ.
Here is how I implemented it:
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
final SSHClient ssh = new SSHClient();
ssh.loadKnownHosts();
ssh.connect("serverName");
try{
ssh.authPublickey("myUserId");
final Session session = ssh.startSession();
try{
final Command cmd = session.exec("net send myMachineName Hello!!!");
System.out.println(cmd.getOutputAsString());
System.out.println("\n Exit Status: "+cmd.getExitStatus());
}finally{
session.close();
}
}finally{
ssh.disconnect();
}
}
}
But I get the following exception:
Exception in thread "main" java.io.IOException: Could not load known_hosts
at net.schmizz.sshj.SSHClient.loadKnownHosts(SSHClient.java:528)
at SSHTEST.main(SSHTEST.java:25)
What am I doing wrong?

Use the folowing code
final SSHClient ssh = new SSHClient();
ssh.addHostKeyVerifier(
new HostKeyVerifier() {
public boolean verify(String arg0, int arg1, PublicKey arg2) {
return true; // don't bother verifying
}
}
);
ssh.connect("LocalHost");

Remove the call to loadKnownHosts() method, which as erickson mentioned checks under ~/.ssh/known_hosts by default (you can specify the location as an argument as well though), and replace it with:
ssh.addHostKeyVerifier("public-key-fingerprint");
To find out what the fingerprint is, the twisted way would be to connect without that statement - you'll find out from the exception ;-)

It sounds like it's trying to read a "known_hosts" file, but can't find it, or possibly it in an invalid format.
The SSH known hosts file records the public key for various hosts to thwart some spoofing attacks. Normally it resides in ~/.ssh/known_hosts. Try creating an empty file there and see if that satisfies the library.
The library documentation is likely to address the necessary configuration files.