I want to configure Burp as a proxy for my java code, to see requests and responses.
Burp works fine as a proxy between a web browser, but it doesn't for java application.
I've added to my code such lines:
WebClient client = new WebClient();
System.setProperty("https.proxyHost", "127.0.0.1");
System.setProperty("https.proxyPort", "8081");
client.getOptions().setCssEnabled(false);
client.getOptions().setJavaScriptEnabled(false);
client.getCookieManager().setCookiesEnabled(true);
URL url = new URL("https://www.google.com");
WebRequest request = new WebRequest(url, HttpMethod.GET);
Page page = client.getPage(request);
And configured Burp to listen on 8081 port.
Should I do anything else?
Where it is working for plain Java as
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8081");
URL url = new URL("http://example.com");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
con.getHeaderFields()
.forEach((key, value) -> System.out.printf("%s: %s%n", key, value));
con.disconnect();
you need to configure the proxy differently for htmlunit. One possible way would be to set it with webClientOptions.setProxyConfig(proxyConfig)
WebClient client = new WebClient();
ProxyConfig proxyConfig = new ProxyConfig("127.0.0.1", 8081);
WebClientOptions options = client.getOptions();
options.setProxyConfig(proxyConfig);
URL url = new URL("http://example.com");
WebRequest request = new WebRequest(url, HttpMethod.GET);
Page page = client.getPage(request);
page.getWebResponse().getResponseHeaders().forEach(System.out::println);
client.close();
Related
I am writing a program that needs to access the cPanel API (https://api.docs.cpanel.net). To use the API I need to be authenticated and have the cPanel session id so that I can build my URL. For example to add a new FTP user I would use the following URL - https://hostname.example.com:2083/cpsess##########/execute/Ftp/add_ftp?user=username
When authenticating in cPanel in a browser, the user is redirected to the cPanel home page and the cPanel session id is displayed in the URL.
I am using Apache HTTP client version 4.5.13. I have no problem authenticating using the following code:
HttpHost targetHost = new HttpHost(domain, 2083, "https");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(user, password));
AuthCache authCache = new BasicAuthCache();
authCache.put(targetHost, new BasicScheme());
// Add AuthCache to the execution context
HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache);
// disable the redirect
//HttpClient client = HttpClientBuilder.create().disableRedirectHandling().build();
// dont disable the redirect
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = client.execute(
new HttpGet(url), context);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println(statusCode);
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println(header.toString());
}
HttpEntity entity = response.getEntity();
if (entity != null) {
// return it as a String
String result = EntityUtils.toString(entity);
System.out.println(result);
}
However when I print out my headers I do not have a Location header so I can not get the URL that contains the session Id.
If I disable the redirect using the line that is commented out above:
HttpClient client = HttpClientBuilder.create().disableRedirectHandling().build();
Then I do get a Location header. However it only provides me with the end of the URl like:
Location: /frontend/paper_lantern/index.html
I need the full URL like:
https://someURL.com:2083/cpsess3886765014/frontend/paper_lantern/index.html
So that I can get the cPanel session Id (cpsess3886765014) in the above URL.
Any help would be much appreciated. Thanks!
OK I figured this out. I can use the following code:
List<URI> redirectLocations = context.getRedirectLocations();
for (URI uri : redirectLocations){
System.out.println(uri.toASCIIString());
}
And I can get the redirects
I'm trying to make HTTPS requests through a proxy. Here's what I've got so far, based on code from this question:
try {
HttpsURLConnection connection = (HttpsURLConnection) new URL("https://proxylist.geonode.com/api/proxy-list?limit=1&page=1&sort_by=speed&sort_type=asc&protocols=https").openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestProperty("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36");
connection.setConnectTimeout(30000);
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String rawJSON = reader.readLine();
if(rawJSON == null) throw new IOException("No data");
JSONObject data = new JSONObject(rawJSON).getJSONArray("data").getJSONObject(0);
String ipAddress = data.getString("ip"), port = data.getString("port");
System.setProperty("https.proxyHost", ipAddress);
System.setProperty("https.proxyPort", port);
SSLContext sslContext = SSLContext.getInstance("SSL");
// set up a TrustManager that trusts everything
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) {}
public void checkServerTrusted(X509Certificate[] certs, String authType) {}
} }, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier((arg0, arg1) -> true);
HttpsURLConnection testConnection = (HttpsURLConnection) new URL("https://example.com").openConnection();
testConnection.connect();
StringBuilder result = new StringBuilder();
String line;
try(BufferedReader reader2 = new BufferedReader(new InputStreamReader(testConnection.getInputStream()))) {
while ((line = reader2.readLine()) != null) result.append(line);
}
System.out.println(result);
} catch(Exception e) {
e.printStackTrace();
}
The code works, but there's a problem. My application (https://encyclosearch.org) is multithreaded, and I need to make some requests through a proxy, and some directly. Since system properties are global, if I set https.proxyHost and https.proxyPort using System.setProperty, some requests that aren't supposed to go through the proxy will go through the proxy.
I can use java.net.Proxy like this:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ipAddress, Integer.parseInt(port)));
HttpsURLConnection testConnection = (HttpsURLConnection) new URL("http://example.com").openConnection(proxy);
But that only works for HTTP proxies, not HTTPS ones, so I can't make HTTPS requests. There's no Proxy.Type.HTTPS.
Any help would be greatly appreciated. Thanks in advance.
If you choose which connections go through a proxy and which do not by the destination url then you can use the property http.nonProxyHosts. This property is used for http and https as the documentation states:
for the "non proxy hosts" list, the HTTPS protocol handler will use the same as the http handler (i.e. http.nonProxyHosts).
You set the property value by adding patterns of urls separated by | For example:
System.setProperty("http.nonProxyHosts", ”localhost|host.example.com”)
You could also use ProxySelector class for choosing which connections go through a proxy. for more info (a bit old):
https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html
As for the absence of Proxy.Type.HTTPS, it is because a proxy in general is not the final destination so the secure connection will be with the final destination not the proxy itself. There is such thing as SSL tunneling through a proxy, but i am not well informed about it.
With #Bashi's help, I figured it out. For direct connections, I used:
url.openConnection(Proxy.NO_PROXY);
This works for Jsoup, too:
Document document = Jsoup.connect("https://example.com").proxy(Proxy.NO_PROXY).get();
Explanation from https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html:
Now, this guarantees you that this specific URL will be retrieved though a direct connection bypassing any other proxy settings, which can be convenient.
I have the following code, but when I run it I get an exception
"SocketTimeoutException" at openStream.
Code:
String urlStr = "https://www.nse-india.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=SCHNEIDER&series=EQ&fromDate=01-01-2020&toDate=29-02-2020&datePeriod=&hiddDwnld=true";
URL urlConn = new URL(urlStr);
InputStream in = urlConn.openStream();
When I execute the same URL from browser, it works fine.
The server looks for two request headers, the below code works
String urlStr = "https://www.nse-india.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=SCHNEIDER&series=EQ&fromDate=01-01-2020&toDate=29-02-2020&datePeriod=&hiddDwnld=true";
URL url = new URL(urlStr);
URLConnection conn = url.openConnection();
conn.setRequestProperty("accept-language", "en-US,en;q=0.9");
conn.setRequestProperty("user-agent", "MyJavaApp");
InputStream in = conn.getInputStream();
When I execute the same URL from browser, it works fine.
There is obviously a difference in what your browser does and what your JVM does. I guess that your browser has a HTTP proxy server configured, but your application hasn't?
i have a secured url , if i open in browser, a pop up comes and i authenitcate with userid/password, then a pdf file is downloaded .
I want this functionality in java. It should authenticate using proxy server/port and user details then download
URL server = new URL("http://some.site.url/download.aspx?client=xyz&docid=1001");
System.setProperty("https.proxyUser", userName);
System.setProperty("https.proxyPassword", password);
System.setProperty("https.proxyHost",proxy);
System.setProperty("https.proxyPort",port);
URLConnection connection = (URLConnection)server.openConnection();
connection.connect();
InputStream is = connection.getInputStream();
//then i read this input stream and write to a pdf file in temporary folder
It gives me connection timeout error.
Then i thought adding authentication
String authentication = "Basic " + new
sun.misc.BASE64Encoder().encode("myuserid:mypassword".getBytes());
connection.setRequestProperty("Proxy-Authorization", authentication);
Still doesnt work,
Please let me know .
I solved this issue.
I used a customized authenticator before connecting the URL, and it authenticates and downloads the document. FYI - once connected, till next server restart, it doesn't need authentication.
URL server = new URL(url); //works for https and not for http, i needed https in my case.
Authenticator.setDefault((new MyAuthenticator()));
URLConnection connection = (URLConnection)server.openConnection();
connection.connect();
InputStream is = connection.getInputStream();
.... //write code to fetch inputstream
Define your own authenticator as given below
public class MyAuthenticator extends Authenticator {
final PasswordAuthentication authentication;
public MyAuthenticator(String userName, String password) {
authentication = new PasswordAuthentication(userName, password.toCharArray());
}
}
I deploy rest web-service (Jersey) in embedded jetty.
My server:
Map<String,Object> initMap = new HashMap<String, Object>();
initMap.put("com.sun.jersey.api.json.POJOMappingFeature", "true");
initMap.put("com.sun.jersey.config.property.packages", "com.extern.rest");
ResourceConfig rc = new PackagesResourceConfig(initMap);
restServer = HttpServerFactory.create("http://localhost:5555/core-gw-rs/", rc);
restServer.start()
My client:
URL url = new URL(buildUrl(params, path));
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.connect();
When I do a query from a client, in response I receive a status of 404. When a request is made by a browser to the same address get required result (200 status).
Update
Method:
#GET
#Path("{accountUuid}/domain/rfc822")
public Response fetchData(#PathParam("accountUuid") String accountUuid) {
return Response.ok().build();
}
The problem was that I started another soap service on another server (the same port, but different context) It remains a mystery why I was able to get through the browser.