We Keep Coding

Java URL openStream throws exception

I have the following code, but when I run it I get an exception
"SocketTimeoutException" at openStream.
Code:
String urlStr = "https://www.nse-india.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=SCHNEIDER&series=EQ&fromDate=01-01-2020&toDate=29-02-2020&datePeriod=&hiddDwnld=true";
URL urlConn = new URL(urlStr);
InputStream in = urlConn.openStream();
When I execute the same URL from browser, it works fine.

The server looks for two request headers, the below code works
String urlStr = "https://www.nse-india.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=SCHNEIDER&series=EQ&fromDate=01-01-2020&toDate=29-02-2020&datePeriod=&hiddDwnld=true";
URL url = new URL(urlStr);
URLConnection conn = url.openConnection();
conn.setRequestProperty("accept-language", "en-US,en;q=0.9");
conn.setRequestProperty("user-agent", "MyJavaApp");
InputStream in = conn.getInputStream();

When I execute the same URL from browser, it works fine.
There is obviously a difference in what your browser does and what your JVM does. I guess that your browser has a HTTP proxy server configured, but your application hasn't?

theMovieDB api call throwing IOEXception

I am making an android app that uses theMovieDB API.
Look at the part of my class extending AsyncTask.
private HttpURLConnection urlconnection = null;
private URL url;
protected String doInBackground(String[] task)
{
String DATA=null;
String baseAddress="https://api.themoviedb.org/3/movie/";
String apiKey="225b36fd29826b4c9821dd90bfc4e055";
Uri Url = Uri.parse(baseAddress).buildUpon().appendEncodedPath(task[0]).appendQueryParameter("api_key",apiKey).build();
Log.d("built URL",Url.toString());
try
{
url= new URL(Url.toString());
urlconnection= (HttpURLConnection) url.openConnection();
urlconnection.setRequestMethod("GET");
urlconnection.connect();
InputStream inputStream = urlconnection.getInputStream();
if (inputStream==null)
{
return null;
}
BufferedReader reader= new BufferedReader(new InputStreamReader(inputStream));
StringBuffer buffer=null;
String line;
while ((line=reader.readLine())!=null)
{
buffer.append(line+'\n');
}
DATA=buffer.toString();
}
I am getting IOException (seen in logcat). I checked the built URL on the browser(it was working). The Same set of syntax did work on openweather api. Is there any other thing that themovieDb API need? Help me Solve it. I did check there documentation but there was no info for android.

i got the Solution. I was connected to my mobile hotspot which due to some reason does not work as expected. Switching to my home WIFI fixed the issue.
Thanks for giving your time on my question

How to login to server and then hit a url on the same server to get JSON in return

Whenever I try to hit a url using java it will redirect me to login page. How can I first login then hit a specific url to get JSON in return ?
Here what I tried so far:
try {
URL url = new URL(GET_EXPENSE_FOR_VENDOR_URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
String response;
System.out.println("Output from Server .... \n");
while ((response = br.readLine()) != null) {
System.out.println(response);
Gson gson = new Gson();
gson.fromJson(response, ExpenseAllocationDTO[].class);
Type collectionType = new TypeToken<Collection<ExpenseAllocationDTO>>() {
}.getType();
expenseAllocationList = gson.fromJson(response, collectionType);
expenseAllocationDTODataModel = (new ExpenseAllocationDTODataModel(expenseAllocationList));
if (expenseAllocationList.isEmpty() || expenseAllocationList == null) {
expenseExists = true;
}
conn.disconnect();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

The problem
I believe the initial request is missing some headers. Only the Accept header is set.
How to solve it ?
Option #1
In order to discover the missing headers, open your favorite browser and browse to GET_EXPENSE_FOR_VENDOR_URL. Before browsing, open the webdeveloper toolbar in order to see headers sent by the browser.
Here is a sample screenshot of the webdeveloper toolbar under Chrome on Windows.
.
Option #2
If your browser doesn't have such a toolbar, you can use a tool like Fiddler for finding the missing headers.
Option #3
You can also use a tool like hurl.it in order to test the headers expected by the target server as you discover them. IMO, this tool can be more straight forward than Fiddler during the debugging phase.
Get back to your code
Once you have identified the missing headers, add them to your Java code like this:
URL url = new URL(GET_EXPENSE_FOR_VENDOR_URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Missing-Header-1", "...");
conn.setRequestProperty("Missing-Header-2", "...");

Check if URL is HTTPS or HTTP protocol?

I am currently using the following to read a file from android docs here and here. The user selects (in the settings screen) if their site uses HTTP or HTTPS protocol. If their website uses the HTTP protocol then it works for both HttpURLConnection and HttpsURLConnection, but if their site uses HTTPS protocol then it doesn't work for HttpURLConnection protocol and worst of all it doesn't give me an exception error. Below is the sample code that I am using.
So in essence, how can I check to see if the web url is HTTPS protocol so checking if the user selected the correct protocol?
InputStream inputStream;
HttpURLConnection urlConnection;
HttpsURLConnection urlHttpsConnection;
boolean httpYes, httpsYes;
try {
if (httpSelection.equals("http://")) {
URL url = new URL(weburi);
urlConnection = (HttpURLConnection) url.openConnection();
inputStream = new BufferedInputStream((urlConnection.getInputStream()));
httpYes = True;
}
if (httpSelection.equals("https://")) {
URL url = new URL(weburi);
urlHttpsConnection = (HttpsURLConnection) url.openConnection();
urlHttpsConnection.setSSLSocketFactory(context.getSocketFactory());
inputStream = urlHttpsConnection.getInputStream();
https=True;
}
catch (Exception e) {
//Toast Message displays and settings intent re-starts
}
finally {
readFile(in);
if(httpYes){
urlConnection.disconnect();
httpYes = False;
}
if(httpsYes){
urlHttpsConnection.disconnect();
httpsYes = False;
}
}
}
EDIT:
To elaborate some more. I need to see if it returns a valid response from a website? So if the user selected http instead of https how can I check to see if http is the incorrect prefix/protocol?
How can I check if the website uses HTTPS or HTTP protocol? If the user then only puts in say www.google.com and I append https:// or http:// prefix to it, how do I know which one is the correct one to use?

You can Use android URLUtil to check whether url is HTTP or HTTPS:
public static boolean isHttpUrl (String url)
Returns True iff the url is an http: url.
public static boolean isHttpsUrl (String url)
Returns True iff the url is an https: url.
Edit:
public static boolean isValidUrl (String url)
Returns True iff the url is valid.

URLConnection result = url.openConnection();
if (result instanceof HttpsURLConnection) {
// https
}
else if (result instanceof HttpURLConnection) {
// http
}
else {
// null or something bad happened
}

Try this code:
mConnexion = (URLUtil.isHttpsUrl(mStringUrl)) ? (HttpsURLConnection) url.openConnection() : (HttpURLConnection) url.openConnection();

I have checked URLUtil class and checked that its contain so many methods for these kind of stuff, but i just extend the answer that you can simply do as below also :-
public static boolean isHttpOrHttpsUrl(String url) {
return url.matches("^(http|https|ftp)://.*$");
}

This can be checked by using Util.
isHttpUrl returns True iff the url is an http: url.
isHttpsUrl returns True iff the url is an https: url.

you can try this
boolean httpYes, httpsYes;
try {
URL url = new URL(weburi);
urlConnection = (HttpURLConnection) url.openConnection();
inputStream = new BufferedInputStream((urlConnection.getInputStream()));
httpYes = True;
}
catch (Exception e) {
//Toast Message displays and settings intent re-starts
URL url = new URL(weburi);
urlHttpsConnection = (HttpsURLConnection) url.openConnection();
urlHttpsConnection.setSSLSocketFactory(context.getSocketFactory());
inputStream = urlHttpsConnection.getInputStream();
https=True;
}

I think this works, at least it seems to be working, what do you guys think? I place this if statement just before the httpsYes = True and httpYes = True.
It seems that when the HTTPS protocol is selected it wants to redirect using response code 302, but for all other instances it connects with response code 200. I throw a new ConnectionException() error as that takes the user back to the settings screen to correct the URL error.
For the HTTPS protocol:
if (httpsURLConnection.getResponseCode() != 200) {
throw new ConnectException();
}
For the HTTP protocol:
if (urlConnection.getResponseCode() != 200) {
throw new ConnectException();
}
Comments? Should I use urlConnection.getResponseCode() > 199 && < 300? To cover all successful connects?

My recommendation is created function expression regular.
for example:
public void testUrl(Object urlHttp){
String url = "https://www.google.com";
if(url.matches("^(https?)://.*$")){
Object o = (HttpsURLConnection) urlHttp;
}else{
Object o = (HttpURLConnection) urlHttp;
}
}
Regards

How to validate XML with schema URLs that return HTTP 301?

I need to validate a XML file whose schema URLs (referenced at schemaLocation attribute) return an HTTP 301 error pointing out the correct URL. Moreover, those schemas have schema imports whose URLs return HTTP 301 pointing out the correct URL too.
JDom2 seems unable to resolve the HTTP 301 errors like a simple browser does. Is there a way to force JDom2 to resolve properly such an error?. What JDom2 class/method i must override to do that?. Is there another java XML library that can do that?
FYI: The erroneous URLs are like http://schema-url and the HTTP 301 errors return URLs like
----------------------- some progresses later ------------------------------
A first approach to solve to this problem could be something like this:
JDOM2 - Follow Redirects (HTTP Error 301)
i.e, using the folling snippet:
URL httpurl = new URL(.....);
HTTPURLConnection conn = (HTTPUrlConnection)httpurl.openConnection();
conn.setInstanceFollowRedirects(true);
conn.connect();
Document doc = saxBuilder.build(conn.getInputStream());
But in my case, the application reads the xml from a gzip xml file, so it will not work. I have decided use something like:
final SAXBuilder builder = new SAXBuilder(XMLReaders.XSDVALIDATING);
builder.setEntityResolver(new RedirectEntityResolver());
document = builder.build(isXml);
Where isXml is the decompressed XML file input stream and RedirectEntityResolver is coded like:
public class RedirectEntityResolver implements EntityResolver {
public InputSource resolveEntity(String publicId, String systemId) throws SAXException, IOException {
URL httpurl = new URL(systemId);
HttpURLConnection conn = (HttpURLConnection) httpurl.openConnection();
conn.setInstanceFollowRedirects(true);
conn.connect();
return new InputSource(conn.getInputStream());
}
}
But it doesn't work. The HttpURLConnection seems unable to resolve the redirects and i get the same error:
org.xml.sax.SAXParseException: s4s-elt-character: Non-whitespace characters are not allowed in schema elements other than 'xs:appinfo' and 'xs:documentation'. Saw 'Document Moved'.

The problem is that conn.setInstanceFollowRedirects(true); is not working. View URLConnection Doesn't Follow Redirect anwsers
Instead, using an EntityResolver like the following, it runs perfectly:
public class RedirectEntityResolver implements EntityResolver {
public InputSource resolveEntity(String publicId, String systemId) throws SAXException, IOException {
URL obj = new URL(systemId);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
int status = conn.getResponseCode();
if ((status != HttpURLConnection.HTTP_OK) &&
(status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)) {
String newUrl = conn.getHeaderField("Location");
conn = (HttpURLConnection) new URL(newUrl).openConnection();
}
return new InputSource(conn.getInputStream());
}
}