We Keep Coding

Downloading binary file from url

I am using this code to download files from a url:
FileUtils.copyURLToFile(url, new File("C:/Songs/newsong.mp3"));
When I create the url using for instance,
"https://mjcdn.cc/2/282676442/MjUgU2FhbCAtIFZlZXQgQmFsaml0Lm1wMw==",
this works just fine and the mp3 is downloaded.
However,
if I use another url:
"https://dl.jatt.link/hd.jatt.link/a0339e7c772ed44a770a3fe29e3921a8/uttzv/Hummer-(Mr-Jatt.com).mp3",
the file is 0kb.
I am able to download files from both these urls from within a web browser.
What's wrong here, and how can I fix it.

I noticed a difference between your 2 URLs:
The first one just gives back the file without redirection.
But the second one responds with a redirect (HTTP/1.1 302 Moved Temporarily). It's also a special case, because it's a redirect from HTTPS to HTTP protocol.
Browsers can follow redirects, but your program - for some reason (see below) - can't.
I suggest you to use a HTTP client library (e.g. Apache HTTP client or Jsoup), and configure it to follow redirects (if they don't do it by default).
For example, with Jsoup, you would need a code like this:
String url = "https://dl.jatt.link/hd.jatt.link/a0339e7c772ed44a770a3fe29e3921a8/uttzv/Hummer-(Mr-Jatt.com).mp3";
String filename = "C:/Songs/newsong.mp3";
Response r = Jsoup.connect(url)
//.followRedirects(true) // follow redirects (it's the default)
.ignoreContentType(true) // accept not just HTML
.maxBodySize(10*1000*1000) // accept 10M bytes (default is 1M), or set to 0 for unlimited
.execute(); // send GET request
FileOutputStream out = new FileOutputStream(new File(filename));
out.write(r.bodyAsBytes());
out.close();
Update on #EJP's comment:
I looked up Apache Commons IO's FileUtils class on GitHub. It calls openStream() of the received URL object.
openStream() is a shorthand for openConnection().inputStream().
openConnection() returns an URLConnection object. If there is an appropriate subclass for the protocol used by URL, it will return an instance of that subclass. In this case that's a HttpsURLConnection which is the subclass of HttpURLConnection.
The followRedirects option is defined in HttpURLConnection and it's indeed true by default:
Sets whether HTTP redirects (requests with response code 3xx) should be automatically followed by this class. True by default.
So OP's approach would normally work with redirects too, but it seems that redirection from HTTPS to HTTP is not handled (properly) by HttpsURLConnection. - It's the case that #VGR mentioned in the comments below.
It's possible to handle redirects manually by reading the Location header with HttpsURLConnection, then use it in a new HttpURLConnection. (Example) (I wouldn't be surprised if Jsoup did the same.)
I suggested Jsoup because it already implements a way to handle HTTPS to HTTP redirections correctly and also provides tons of useful features.

Http request returning different response in browser

I'm getting a JSON object from a server, When I enter the following generated URL into my browser I get a response with "num_match": 18, however when running in my app I get a JSON object with "num_matches": 2.
The URL object is created like this
URL request;
request = new URL(url);
and connection like this:
HttpURLConnection connection = (HttpURLConnection) request.openConnection();
connection.setConnectTimeout(MAX_TIME);
connection.setReadTimeout(MAX_TIME);
url is a String and I am copying the string contents into my browser to test.
The string is:
http://search.3taps.com/?auth_token=xxxxxxxxxxxxxxxxxx&retvals=heading,body,timestamp,external_url,images,price&rpp=100&source=BKPGE|CRAIG|EBAYC|INDEE|KIJIJ&category=PWSM&radius=200mi&lat=26.244&long=-80.2&annotations={age:18 OR age:19 OR age:20 OR age:21 OR age:22}
The URL object has the following fields
query:
auth_token=xxxxxxxxxxxxxxxxxx&retvals=heading,body,timestamp,external_url,images,price&rpp=100&source=BKPGE|CRAIG|EBAYC|INDEE|KIJIJ&category=PWSM&radius=200mi&lat=26.244&long=-80.2&annotations={age:18 OR age:19 OR age:20 OR age:21 OR age:22}
file:
/?auth_token=xxxxxxxxxxxxxxxxxx&retvals=heading,body,timestamp,external_url,images,price&rpp=100&source=BKPGE|CRAIG|EBAYC|INDEE|KIJIJ&category=PWSM&radius=200mi&lat=26.244&long=-80.2&annotations={age:18 OR age:19 OR age:20 OR age:21 OR age:22}
host:
search.3taps.com
The response comes back as "success":true on both but with a discrepancy in the object returned. I don't know much about http, what could be causing this?
UPDATE: On further testing it seems like there is only a problem when the annotations entry is present
annotations={age:18 OR age:19 OR age:20 OR age:21 OR age:22}
seems to be causing the problem.

Make sure you are encoding the URL request correctly when you are setting the URL for the server. The spaces, braces, and colons all need to be appropriately escaped. Spaces should be %20, etc. This may help: HTTP URL Address Encoding in Java
Old Answer.... Comments indicate this does not affect the result... so moving down.
It is quite possible that the server is changing it's behaviour based on the type of 'browser' you are reporting yourself to be. When connecting to an HTTP server you tell the server what your UserAgent is (typically for a browser it is something like "Internet Explorer ...." or "Mozilla ..." or "Google Chome ...". The Server will often tailor the results of a request to suite the User Agent (different javascript files and HTML codes go to IE, etc.). This is also how servers re-direct mobile devices to a mobile-friendly version of a site.
It is quite possible that the server is changing it's response to match your UserAgent exposed by your Java code, (which by decault is something like "Java/1.7.0". You can change this value a few ways. Have a look at this question Setting user agent of a java URLConnection and try to run your program with the Mozilla agent, and see if you get different results.

Open an authenticated image served by django from java using Apache http client

I Am serving an authenticated image using django. The image is behind a view which require login, and in the end I have to check more things than just the authentication.
Because of a reason to complicated to explain here, I cannot use the real url to the image, but I Am serving it with a custom url leading to the authenticated view.
From java the image must be reachable, to save or display. For this part I use Apache httpclient.
In Apacahe I tried a lot of things (every example and combination of examples...) but can't seem to get it working.
For other parts of the webapp I use django-rest-framwork, which I succesfully connected to from java (and c and curl).
I use the login_reuired decorator in django, which makes the attempt to get to the url redirect to a login page first.
Trying the link and the login in a webviewer, I see the 200 code (OK) in the server console.
Trying the link with the httpclient, I get a 302 Found in the console.... (looking up 302, it means a redirect..)
this is what I do in django:
in urls.py:
url(r'^photolink/(?P<filename>.*)$', 'myapp.views.photolink',name='photolink'),
in views.py:
import mimetypes
import os
#login_required
def photolink(request, filename):
# from the filename I get the image object, for this question not interesting
# there is a good reason for this complicated way to reach a photo, but not the point here
filename_photo = some_image_object.url
base_filename=os.path.basename(filename_photo)
# than this is the real path and filename to the photo:
path_filename=os.path.join(settings.MEDIA_ROOT,'photos',mac,base_filename)
mime = mimetypes.guess_type(filename_photot)[0]
logger.debug("mimetype response = %s" % mime)
image_data = open(path_filename, 'rb').read()
return HttpResponse(image_data, mimetype=mime)
by the way, if i get this working i need another decorator to pass some other tests....
but i first need to get this thing working....
for now it's not a secured url.... plain http.
in java i tried a lot of things... using apache's httpclient 4.2.1
proxy, cookies, authentication negociation, with follow redirects... and so on...
Am I overlooking some basic thing here?...
it seems the login of the website client is not suitable for automated login...
so the problem can be in my code in django....or in the java code....

In the end the problem was, using HTTP authorization.
Which is not by default used in the login_required decorator.
adding a custom decorator that checks for HTTP authorization did the trick:
see this example: http://djangosnippets.org/snippets/243/

Access URL with double slash with HttpClient

I'm doing a request on another server like this:
HttpGet req = new HttpGet("http://example.com//foo");
new DefaultHttpClient().execute(req);
However, HttpClient changes example.com//foo to example.com/foo, so the other server (which is not mine) doesn't understand the request.
How can I fix this?

A double-slash is not a legal in the path section of a URI (see RFC2396, sections 3.2, 3.4). The '//' sequence has a defined meaning in the URI, it denotes the authority component (server).
I realize this does not answer your question but the HttpClient is, in fact, behaving in accordance with the HTTP and URL standards. The server your are reading from is not. This appears to be previously reported (https://issues.apache.org/jira/browse/HTTPCLIENT-727) and discarded by the HttpClient team.

It is an illegal URL in fact.
Did you try passing an URI instead of a String?
Did you try / \ \ / ? Or the URL might be equivalent to /default.asp/, /index.html/, /./, /?/, example.com/foo/ or the like.
Otherwise you will need to hack the sources.

I also wanted to do same thing and Apache Http client don't support that.
I managed to get it done using a Netty. I wrote http client using Netty and with that I was able send request with double slash(//) in the path. I used SnoopClient as sample.

Acegi throws AuthenticationCredentialsNotFoundException when opening URl with BrowserLauncher 2

We have a JSF web application that uses Acegi security. We also have a standalone Java Swing application. One function of the Swing app is to load the user's home page in a browser window.
To do this we're currently using Commons HttpClient to authenticate the user with the web app:
String url = "http://someUrl/j_acegi_security_check";
HttpClient client = new HttpClient();
System.setProperty(trustStoreType, "Windows-ROOT");
PostMethod method = new PostMethod(url);
method.addParameter("j_username", "USERNAME");
method.addParameter("j_password", "PASSWORD");
int statusCode = client.executeMethod(method);
if (statusCode == HttpStatus.SC_MOVED_TEMPORARILY ) {
Header locationHeader= method.getResponseHeader("Location");
String redirectUrl = locationHeader.getValue();
BrowserLauncher launcher = new BrowserLauncher();
launcher.openURLinBrowser(redirectUrl);
}
This returns a HTTP 302 redirect response, from which we take the redirect url and open it using BrowserLauncher 2. The url contains the new session ID, something like:
http://someUrl/HomePage.jsf;jsessionid=C4FB2F643CE48AC2DE4A8A4C354033D4
The problem we're seeing is that Acegi processes the redirect but throws an AuthenticationCredentialsNotFoundException. It seems that for some reason the authenticated credentials cannot be found in the security context.
Does anyone have an idea as to why this is happening? If anyone needs more info then I'll be happy to oblige.
Many thanks,
Richard

I have never done Acegi/SpringSecurity, but the symptoms are clear enough: some important information is missing in the request. You at least need to investigate all the response headers if there isn't something new which needs to be passed back in the header of the subsequent request. Maybe another cookie entry which represents the Acegi credentials.
But another caveat is that you in fact cannot open just the URL in a local browser instance, because there's no way to pass the necessary request headers along it. You'll need to have your Swing application act as a builtin webbrowser. E.g. get HTML response in an InputStream and render/display it somehow in a Swing frame. I would check if there isn't already an existing API for that, because it would involve much more work than you'd initially think .. (understatement).

In this case you can do Basic Authentication and set this header in every request instead of sending the jsessionid:
AUTHORIZATION:Basic VVNFUk5BTUU6UEFTU1dPUkQ=
The token VVNFUk5BTUU6UEFTU1dPUkQ= is the username and the password encoded base64.
Example:
scott:tiger
is:
c2NvdHQ6dGlnZXI=
One more thing: use SSL.