We Keep Coding

Read Fortnite Json API

So I'm trying to read a JSON file in from a website (fortniteapi.com), every time I try to download the file to my local computer it does not download. I've been at this for about a week and I just can't figure out why it won't work.
also i'm using Gson
Here is my code so far:
package sample;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.stage.Stage;
import java.io.*;
import java.net.URL;
import java.net.URLConnection;
public class Main extends Application {
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Fortnite");
primaryStage.setScene(new Scene(root, 300, 275));
primaryStage.show();
ReadJson();
}
public static void main(String[] args) {
launch(args);
}
public void ReadJson()
{
try {
// read url
String sURL = "https://fortnite-public-api.theapinetwork.com/prod09/users/id?username=Ninja"; //just a string
// Connect to the URL using java's native library
URL url = new URL(sURL);
URLConnection request = url.openConnection();
request.connect();
// Convert to a JSON object
JsonParser jp = new JsonParser(); //from gson
JsonElement root = jp.parse(new InputStreamReader((InputStream) request.getContent())); //Convert the input stream to a json element
JsonObject rootobj = root.getAsJsonObject();
String output = rootobj.get("username").getAsString(); //just grab the username value
// print out the result/output
System.out.println(output);
} catch (IOException e) {
System.out.println("Unexpected Error.");
// JOptionPane.showMessageDialog(null, "Oh no something went wrong.", "Unexpected Error", JOptionPane.ERROR_MESSAGE);
System.exit(1);
}
}
}

The error
After reading the errorStream() of the request (after casting it to HttpURLConnection) HTML is printed and states:
Access denied | fortnite-public-api.theapinetwork.com used Cloudflare
to restrict access
and
The owner of this website (fortnite-public-api.theapinetwork.com) has
banned your access based on your browser's signature
(mybrowsersignature).
What does this mean
Cloudflare states that that error means that:
the domain owner is blocking this request based on the client's web
browser signature.
and that the feature is called "Browser Integrity Check", from there we can find What does the Browser Integrity Check do?:
Cloudflare's Browser Integrity Check (BIC) is similar to Bad Behavior
and looks for common HTTP headers abused most commonly by spammers and
denies access to your page. It will also challenge visitors that do
not have a user agent or a non standard user agent (also commonly used
by abuse bots, crawlers or visitors).
Solution
We can change the User-Agent of request to something that should be valid before request.connect(); like so (user agent copied from User-Agent | MDN):
request.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0");
The expected output is printed:
Ninja

Copy of InputStream blocks with Jetty HTTP client using an InputStreamResponseListener

Im am using Jetty 9.4.8 HTTP client and want to write a stream of incoming data to a file. Currently I am using an InputStreamResponseListener and IOUtils.copy(..) writing to a FileOutputStream. I have also tried Files.copy().
InputStreamResponseListener streamResponseListener = new InputStreamResponseListener();
request.send(streamResponseListener);
if(streamResponseListener.get(5, TimeUnit.MINUTES).getStatus() == 200) {
OutputStream outputStream = null;
try {
TMP_FILE.toFile().createNewFile();
outputStream = new FileOutputStream(TMP_FILE.toFile());
IOUtils.copy(inputStream, outputStream);
} catch(IOException e) {
this.getLogService().log(..)
} finally {
IOUtils.closeQuietly(inputStream);
IOUtils.closeQuietly(outputStream);
}
// NOT REACHED IN CASE InputStream is BLOCKED FOR SOME REASON
}
However, the copy methods seem to block after all bytes have been received. Why could this happen and how can I avoid this?
Headers of the HTTP content requested:
Date: Wed, 23 May 2018 16:46:06 GMT
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=".."
Content-Length: 613970044
Server: Jetty(9.4.8.v20171121)
IOUtils from Apache Commons IO Version 2.4

Here's a working example of your codebase, using only Java and Jetty.
This is requesting content from a server that is known to comply with the HTTP spec.
package demo.jettyclient;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.api.Request;
import org.eclipse.jetty.client.api.Response;
import org.eclipse.jetty.client.util.InputStreamResponseListener;
import org.eclipse.jetty.http.HttpStatus;
import org.eclipse.jetty.util.IO;
import org.eclipse.jetty.util.StringUtil;
import org.eclipse.jetty.util.ssl.SslContextFactory;
public class DownloadUrl
{
public static void main(String[] args) throws Exception
{
String uriString = "https://upload.wikimedia.org/wikipedia/en/a/a1/Jetty_logo.png?download";
if (args.length >= 1)
uriString = args[0];
URI srcUri = URI.create(uriString);
SslContextFactory ssl = new SslContextFactory(true);
HttpClient client = new HttpClient(ssl);
try
{
client.start();
Request request = client.newRequest(srcUri);
System.out.printf("Using HttpClient v%s%n", getHttpClientVersion());
System.out.printf("Requesting: %s%n", srcUri);
InputStreamResponseListener streamResponseListener = new InputStreamResponseListener();
request.send(streamResponseListener);
Response response = streamResponseListener.get(5, TimeUnit.SECONDS);
if (response.getStatus() != HttpStatus.OK_200)
{
throw new IOException(
String.format("Failed to GET URI [%d %s]: %s",
response.getStatus(),
response.getReason(),
srcUri));
}
Path tmpFile = Files.createTempFile("tmp", ".dl");
try (InputStream inputStream = streamResponseListener.getInputStream();
OutputStream outputStream = Files.newOutputStream(tmpFile))
{
IO.copy(inputStream, outputStream);
}
System.out.printf("Downloaded %s%n", srcUri);
System.out.printf("Destination: %s (%,d bytes)%n", tmpFile.toString(), Files.size(tmpFile));
}
finally
{
client.stop();
}
}
private static String getHttpClientVersion()
{
ClassLoader cl = HttpClient.class.getClassLoader();
// Attempt to use maven pom properties first
String pomResource = "/META-INF/maven/org/eclipse/jetty/jetty-client/pom.properties";
URL url = cl.getResource(pomResource);
if (url != null)
{
try (InputStream in = url.openStream())
{
Properties props = new Properties();
props.load(in);
String version = props.getProperty("version");
if (StringUtil.isNotBlank(version))
return version;
}
catch (IOException ignore)
{
}
}
// Attempt to use META-INF/MANIFEST.MF
String version = HttpClient.class.getPackage().getImplementationVersion();
if (StringUtil.isNotBlank(version))
return version;
return "<unknown>";
}
}
When run, this results in ...
2018-05-23 10:52:08.401:INFO::main: Logging initialized #325ms to org.eclipse.jetty.util.log.StdErrLog
Using HttpClient v9.4.9.v20180320
Requesting: https://upload.wikimedia.org/wikipedia/en/a/a1/Jetty_logo.png?download
Downloaded https://upload.wikimedia.org/wikipedia/en/a/a1/Jetty_logo.png?download
Destination: C:\Users\joakim\AppData\Local\Temp\tmp2166600286896937563.dl (11,604 bytes)
Process finished with exit code 0
One (or more) of the following is likely causing your issue.
There is either something wrong with your server, not complying with the HTTP spec.
The HTTP exchange isn't complete yet (from the protocol point of view). Capture the traffic and verify the behavior.
The IOUtil library you are using (you didn't say which one) has a bug.
The fact that wget (or curl) works is likely because they are not strict with Content-Length (per recommendations in RFC7230) and will display / download all content received until physical connection EOF/disconnect. While the HTTP/1.1 protocol has a connection persistence and strict rules on when the request (and response) content ends.

Java (JRE 1.8.0_141) - Error 405 for GET request

I am using Java JRE 1.8.0_141 and I am trying to access a specific URL and store the HTML into a String so that I can manipulate the data later in the code, but I keep getting error 405 whenever I call getInputStream().
The code seems to work with other URLs without problems. The trouble URL is:
http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
Here is Eclipse 4.6.3's specific error:
<terminated, exit value: 1>C:\Program Files\Java\jre1.8.0_141\bin\javaw.exe (Aug 6, 2017, 10:53:37 PM)
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at RunMe.getHTMLFromURL(RunMe.java:52)
at RunMe.main(RunMe.java:18)
Caused by: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at RunMe.getHTMLFromURL(RunMe.java:36)
... 1 more
My RunMe.java code is below:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.LinkedList;
public class RunMe {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
System.out.println(getHTMLFromURL("http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true"));
}
public static String getHTMLFromURL(String url){
try{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(false);
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
// CODE FAILS HERE ^
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null){
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e){
throw new RuntimeException(e);
}
}
}
Any idea how I can pull the HTML from this URL if not via this method? Thanks in advance!

I executed a curl command against the URL and it seems like the site is trying to run JavaScript for rendering the page.
curl -v -L -H "User-Agent: Mozilla/5.0" -H "Accept: text/html" "http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2"
> GET /for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2 HTTP/1.1
> Host: www.streeteasy.com
> User-Agent: Mozilla/5.0
> Accept: text/html
>
< HTTP/1.1 405 Not Allowed
// elided
<h1>Pardon Our Interruption...</h1>
<p>As you were browsing <strong>www.streeteasy.com</strong> something about your browser made us think you were a bot. There are a few reasons this might happen:</p>
<ul>
<li>You're a power user moving through this website with super-human speed.</li>
<li>You've disabled JavaScript in your web browser.</li>
<li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a title='Third party browser plugins that block javascript' href='http://ds.tl/help-third-party-plugins' target='_blank'>support article</a>.</li>
</ul>
<p>After completing the CAPTCHA below, you will immediately regain access to www.streeteasy.com.</p>
You may be outta luck here unless you can programmatically fill in the captcha.
Edit:
The problem is clearly cookies as indicated in the discussion below.

Java Rest API client - GET method - Error 415

I am new to writing Java client for Restful API using Apache CXF.
On running below code I am getting error 415 returned which when I looked online shows as "unsupported media type". In order to fix it I changed the code to "target.request(MediaType.APPLICATION_XML)" from original target.request(). However this didn't fix the code.
What is the best way to debug this issue?
Thanks a lot in advance for your time.
Update: After discussion with the Rest API developer I came to know that I need to add a header "("Content-Type", "application/x-www-form-urlencoded");". but I am not sure how to add a header. Does anyone know how to add this header here?
package com.blackhawk.ivr.restAPI.client;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Invocation;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
public class BlissRestAPI {
public static final String BLISS_SERVICRE_URL = "http://x.x.x.x:9090/services";
public static void main(String[] args) {
Client client = ClientBuilder.newClient();
WebTarget target = client.target(BLISS_SERVICRE_URL);
target = target.path("/cardmanagementservices/v3/card/status").queryParam("ani", "xxxxxxxxxx").queryParam("card.expiration", "xxxxxx").queryParam("card.number", "xxxxxxxxxxxxxxxx").queryParam("channel.id", "xyz");
Invocation.Builder builder = target.request(MediaType.APPLICATION_XML);
Response response = builder.get();
System.out.println(response.getStatus());
response.close();
client.close();
}
}

First you can change the media type as given below.
Client: MediaType.APPLICATION_XML
Rest: MediaType.APPLICATION_JSON
JAX-WS are Java standard to build web service. So you have used it here, As my knowledge it is easy to use axis 2 to this kind of web services and clients since there are more implementations of JAX-WS. So i will give you a solution using apache axis technology.
import org.apache.axis.client.Call;
import org.apache.axis.client.Service;
import org.apache.axis.encoding.XMLType;
import javax.xml.rpc.ParameterMode;
public class axisClient {
public static void main(String [] args) throws Exception {
String endpoint = "http://localhost:8090/archive_name/service_name.jws";
Service service = new Service();
Call call = (Call) service.createCall();
call.setTargetEndpointAddress( new java.net.URL(endpoint) );
call.setOperationName( "service_method_name" );
call.addParameter("parameter_name", XMLType.XSD_STRING, ParameterMode.IN );
call.setReturnType( XMLType.XSD_STRING );
call.setProperty(Call.CHARACTER_SET_ENCODING, "UTF-8");
String jsonString = (String) call.invoke( new Object [] { "parameter_value"});
System.out.println("Got result : " + jsonString);
}
}

I got it working by using below code (got 200 status returned)
WebClient client = WebClient.create(BLISS_SERVICRE_URL);
client.path("/cardmanagementservices/v3/card/status").query("ani", "xxxxxxxxxx").query("card.expiration", "xxxxxx").query("card.number", "xxxxxxxxxxxxxx").query("channel.id", "xxxxx");
client.type(MediaType.APPLICATION_FORM_URLENCODED).accept(MediaType.APPLICATION_XML);
client.header("Content-Type","application/x-www-form-urlencoded");
Response response = client.get();
System.out.println(response.getStatus());

Jsoup not downloading entire page

The webpage is: http://www.hkex.com.hk/eng/market/sec_tradinfo/stockcode/eisdeqty_pf.htm
I want to extract all the <tr class="tr_normal"> elements using Jsoup.
The code I am using is:
Document doc = Jsoup.connect(url).get();
Elements es = doc.getElementsByClass("tr_normal");
System.out.println(es.size());
But the size (1350) is smaller than actually have (1452).
I copied this page onto my computer and deleted some <tr> elements. Then I ran the same code and it's correct. It looks like there are too many elements so jsoup can't read all of them?
So what's happened? Thanks!

The problem is the internal Jsoup Http Connection Handling. Nothing wrong with the selector engine. I didn't go deep in but there always problem with proprietary way to handle http connection. I would recommend to replace it with HttpClient - http://hc.apache.org/ . If you can't add http client as dependencies, you might want to check Jsoup source code in handling http connection.
The issue is the default maxBodySize of Jsoup.Connection. Please refer to updated answer. *I still keep HttpClient code as sample.
Output of the program
load from file= 1452
load from http client= 1452
load from jsoup connect= 1350
load from jsoup connect using maxBodySize= 1452
package test;
import java.io.IOException;
import java.io.InputStream;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
public class TestJsoup {
/**
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException {
Document doc = Jsoup.parse(loadContentFromClasspath(), "UTF8", "");
Elements es = doc.getElementsByClass("tr_normal");
System.out.println("load from file= " + es.size());
doc = Jsoup.parse(loadContentByHttpClient(), "UTF8", "");
es = doc.getElementsByClass("tr_normal");
System.out.println("load from http client= " + es.size());
String url = "http://www.hkex.com.hk/eng/market/sec_tradinfo"
+ "/stockcode/eisdeqty_pf.htm";
doc = Jsoup.connect(url).get();
es = doc.getElementsByClass("tr_normal");
System.out.println("load from jsoup connect= " + es.size());
int maxBodySize = 2048000;//2MB (default is 1MB) 0 for unlimited size
doc = Jsoup.connect(url).maxBodySize(maxBodySize).get();
es = doc.getElementsByClass("tr_normal");
System.out.println("load from jsoup connect using maxBodySize= " + es.size());
}
public static InputStream loadContentByHttpClient()
throws ClientProtocolException, IOException {
String url = "http://www.hkex.com.hk/eng/market/sec_tradinfo"
+ "/stockcode/eisdeqty_pf.htm";
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
return response.getEntity().getContent();
}
public static InputStream loadContentFromClasspath()
throws ClientProtocolException, IOException {
return TestJsoup.class.getClassLoader().getResourceAsStream(
"eisdeqty_pf.htm");
}
}