I need to talk to an obscure webserver which requires authentication. If I don't supply credentials, a login form is displayed. However, if I do supply unsolicited Basic Authentication credentials, I get directly to the desired content.
wget supports this directly:
# this fails and downloads a form:
wget https://weird.egg/data.txt --http-user=me --http-password=shhh
# this works and downloads the document:
wget https://weird.egg/data.txt --http-user=me --http-password=shhh --auth-no-challenge
Now my question: How can I make the download in Java using Apache's HTTP Components?
Here's what I got so far. (There's also a proxy in place, and I use -Y on in wget, and I have a matching https_proxy environment variable.)
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import java.net.URI;
// ...
DefaultHttpClient hc = new DefaultHttpClient();
hc.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, new HttpHost(proxy_name, proxy_port));
URI uri = new URI("https://weird.egg/data.txt");
hc..getCredentialsProvider().setCredentials(new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT, AuthScope.ANY_REALM, AuthScope.ANY_SCHEME), new UsernamePasswordCredentials("me", "shh"));
hc.execute(new HttpGet(uri)); // etc
However, I only end up with the login form page, not the actual document. I'm suspecting that the DefaultHttpClient isn't sending the credentials unsolicited, in the way that wget does. Is there a way to make the Java program send the credentials?
Never mind. I solved the problem by not trying to use any library authentication methods, but just brute-forcing the Basic Authentication header into the request:
HttpGet get = new HttpGet(uri);
String basic_auth = new String(Base64.encodeBase64((username + ":" + password).getBytes()));
get.addHeader("Authorization", "Basic " + basic_auth);
hc.execute(get); // etc
(This needs the additional import org.apache.commons.codec.binary.Base64;, but in turn we can remove the credential-related imports.)
Related
I'm trying to get the text response from the following URL:
http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD
The response is the following:
[[["Hello there","你好",,,1]],,"zh-CN"]
(You can verify this response by entering the address into your browser.)
Here is a simplified version of my code that tries to download this text:
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
public class Test {
public static String downloadString() {
String url = "http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD";
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
ResponseHandler<String> handler = new BasicResponseHandler();
try {
return client.execute(request, handler);
} catch (Exception e) {
return "GET request failed.";
}
}
}
When I call Test.downloadString(), I get the following (incorrect) response:
[[["Huan Chai Sunsolt","浣犲ソ",,,0]],,"zh-CN"]
I'm guessing that there is some sort of encoding problem behind the scenes somewhere in the request process (there are six bytes that should be interpreted as two Chinese characters, but are instead interpreted as three Japanese characters), but I can't seem to pinpoint the exact cause. What am I doing wrong in my code?
It's strange, but adding the User-Agent header fixed the problem:
request.addHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0");
Android 6.0 release removes support for the Apache HTTP client. If your app is using this client and targets Android 2.3 (API level 9) or higher, use the HttpURLConnection class instead.
here: http://developer.android.com/about/versions/marshmallow/android-6.0-changes.html#behavior-apache-http-client
I'm trying to login remotely to a Jenkins server using Java.
I didn't found some documentation on how this should be securely done.
For my local server using the url: http://user:pass#server doesn't work .
Can anybody recommend me some documentation regarding this topic ?
Documention from jenkins wiki
Should work with your Basic Authentication type.
Java example with httpclient 4.3.x
import java.io.IOException;
import java.net.URI;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.AuthCache;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
public class JenkinsScraper {
public String scrape(String urlString, String username, String password) throws ClientProtocolException, IOException {
URI uri = URI.create(urlString);
HttpHost host = new HttpHost(uri.getHost(), uri.getPort(), uri.getScheme());
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(new AuthScope(uri.getHost(), uri.getPort()), new UsernamePasswordCredentials(username, password));
// Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// Generate BASIC scheme object and add it to the local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put(host, basicAuth);
CloseableHttpClient httpClient = HttpClients.custom().setDefaultCredentialsProvider(credsProvider).build();
HttpGet httpGet = new HttpGet(uri);
// Add AuthCache to the execution context
HttpClientContext localContext = HttpClientContext.create();
localContext.setAuthCache(authCache);
HttpResponse response = httpClient.execute(host, httpGet, localContext);
return EntityUtils.toString(response.getEntity());
}
}
The Http/Html way is the most cumbersome! I would use jenkins cli or remote api. If you still insist using Java with http then you need to use basic http-authentication and if you plan on trigger more steps inside jenkins use a proper Http/Html Java library like Java-Selenium or HttpUnit.
Best and simple solution for using http basic auth in Java I found here:
Http Basic Authentication in Java using HttpClient?
Also check, if your use case can be covered by jenkins cli:
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI
How to use jenkins cli from the command line:
java -jar jenkins-cli.jar -s yourserver.com help [command]
The cli works with username & password and also can handle certificates for https connection.
Also check the Remote API:
https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
I am using HttpClient (version 3.1) on several different (but apparently identical) computers to read a UTF-8 encoded JSON data from a URL.
On all the machines, save one, it works fine. I have some Spanish language words and they come through with accents and tildes intact.
One computer stubbornly refuses to cooperate. It is apparently treating the data as ISO-8859-1, despite a Content-Type: application/json;charset=utf-8 header.
If I use curl to access that URL from that computer, it works correctly. On every other computer, both curl and my HttpClient-based program work correctly.
I did an md5sum on the common-httpclient.jar file on each machine: the same.
Is there some setting, deep in Linux, that might be different and be messing with me? Any other theories, or even places to look?
EDIT: some people asked for more details.
Originally I had the problem deep in the bowels of a complex Tomcat app, but I lightly adapted the sample to just retrieve the URL in question, and (fortunately) had the same problem.
These are Linux 2.6 machines running jdk1.7.0_45.
An env command yields a bunch of variables. The only one that looks remotely on point is LANG=en_US.UTF-8.
How do you get the json response data from HttpClient?
If you get it back in binary form (through getResponseBodyAsStream() for example), and then convert it to a String without specifying charset, then the result depends on your JVM's default charset.
You can check the value of JVM default charset by:
Charset.defaultCharset().name()
This might give "UTF-8" on all machines except the one failing.
Without seeing your code, it is difficult to say what's wrong, but here is a "correct" way of doing this (using HttpClient 3.1.0 for request and Jackson 2.1.3 to parse the JSON).
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.http.HttpStatus;
import java.io.IOException;
import java.io.InputStreamReader;
HttpClient hc = new HttpClient();
GetMethod get = new GetMethod(uri);
int status = hc.executeMethod(get);
if (status != HttpStatus.SC_OK) throw new RuntimeException("http status " + status);
ObjectMapper jsonParser = new ObjectMapper(new JsonFactory());
// we use an InputStreamReader with explicit charset to read the response body
JsonNode json = jsonParser.readTree(
new InputStreamReader(get.getResponseBodyAsStream(), get.getResponseCharSet())
);
I already faced this issue and this was because of the encoding type configured in the client. So I had to make a "work around" like the one below:
String encmsg = new String(respStr.getBytes("ISO-8859-1"), java.nio.charset.Charset.forName("UTF-8"));
It reads the String as ISO-8859-1 and convert to UTF-8.
How to make a cross-domain call from GWT?
I found JSONPRequestBuilder as a solution, but it can only create GET request not POST. I am trying to call URL shortner service ("http://goo.gl/api/shorten") of google.
From servlet on GAE you can call external http services via URLFetch.
From client side GWT you can directly call Google Shortener API via gwt-google-apis. See the shortener example at the end of page.
Got it through URLFetch. Below is my code:
//Classes to import
import com.google.appengine.api.urlfetch.HTTPMethod;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.HTTPResponse;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;
//Shortening download URL
URL url=new URL("http://goo.gl/api/shorten");
HTTPRequest req=new HTTPRequest(url,HTTPMethod.POST);
req.setPayload(("url=www.google.com").getBytes());
URLFetchService service = URLFetchServiceFactory.getURLFetchService();
HTTPResponse response = service.fetch(req);
byte[] content = response.getContent();
String urlshort=new String(content); //here is the JSON data from goo.gl
im trying to implement a simple comet app to just send data and recive data, my client side is written on java and the server is in node.js, im trying to implement it from the client side with HttpUrlConnection but it seems that when i try to write to the server it doesnt respond me. so how can code the server to respond? (currently using http.createServer(function(req, res){...}).listen(8124);
I'm doing something similar to that with an Android app. I can't say that this is the "right" way to do it, but I'm using these classes in my project (to make POSTs and check the responses):
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
To implement I'm doing something to the effect of
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(/*...(String) url...*/);
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("key", "value"));
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = httpclient.execute(httppost); // <-- this has useful info
To actually get useful stuff from the server I'm using JSON because, IMHO, XML parsing is a huge pain in Java.
So these are the classes I use for parsing the JSON that nodejs spits out at me.
import org.json.JSONArray;
import org.json.JSONObject;
...
JSONObject jObject = new JSONObject(EntityUtils.toString(response.getEntity()));
JSONArray datasets = jObject.getJSONArray("blahblahblah");
Having used it (quite easily) in my own Android app, I recommend Socket.IO-client Java.
It's a "full-featured Socket.IO Client Library for Java, which is compatible with Socket.IO v1.0 and later."