So I'm trying to read a JSON file in from a website (fortniteapi.com), every time I try to download the file to my local computer it does not download. I've been at this for about a week and I just can't figure out why it won't work.
also i'm using Gson
Here is my code so far:
package sample;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.stage.Stage;
import java.io.*;
import java.net.URL;
import java.net.URLConnection;
public class Main extends Application {
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Fortnite");
primaryStage.setScene(new Scene(root, 300, 275));
primaryStage.show();
ReadJson();
}
public static void main(String[] args) {
launch(args);
}
public void ReadJson()
{
try {
// read url
String sURL = "https://fortnite-public-api.theapinetwork.com/prod09/users/id?username=Ninja"; //just a string
// Connect to the URL using java's native library
URL url = new URL(sURL);
URLConnection request = url.openConnection();
request.connect();
// Convert to a JSON object
JsonParser jp = new JsonParser(); //from gson
JsonElement root = jp.parse(new InputStreamReader((InputStream) request.getContent())); //Convert the input stream to a json element
JsonObject rootobj = root.getAsJsonObject();
String output = rootobj.get("username").getAsString(); //just grab the username value
// print out the result/output
System.out.println(output);
} catch (IOException e) {
System.out.println("Unexpected Error.");
// JOptionPane.showMessageDialog(null, "Oh no something went wrong.", "Unexpected Error", JOptionPane.ERROR_MESSAGE);
System.exit(1);
}
}
}
The error
After reading the errorStream() of the request (after casting it to HttpURLConnection) HTML is printed and states:
Access denied | fortnite-public-api.theapinetwork.com used Cloudflare
to restrict access
and
The owner of this website (fortnite-public-api.theapinetwork.com) has
banned your access based on your browser's signature
(mybrowsersignature).
What does this mean
Cloudflare states that that error means that:
the domain owner is blocking this request based on the client's web
browser signature.
and that the feature is called "Browser Integrity Check", from there we can find What does the Browser Integrity Check do?:
Cloudflare's Browser Integrity Check (BIC) is similar to Bad Behavior
and looks for common HTTP headers abused most commonly by spammers and
denies access to your page. It will also challenge visitors that do
not have a user agent or a non standard user agent (also commonly used
by abuse bots, crawlers or visitors).
Solution
We can change the User-Agent of request to something that should be valid before request.connect(); like so (user agent copied from User-Agent | MDN):
request.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0");
The expected output is printed:
Ninja
Related
I currently have an instance of a Document object. The Document object uses JSoup's connect method to fetch a Http Request. When I call the .html() method on the instance doc and print the result, there seems to be missing tags. When comparing my output to the source code on my browser (FireFox), there seems to be missing elements (More specifically, a youtube video for instance will have a div tag with a class attribute of "html5-video-container").
For reference the source code that I am using is as follows:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
public class JsoupTester {
public static void main(String[] args){
try {
Document doc = Jsoup.connect("https://m.youtube.com/watch?v=ycPr5-27vSI").userAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) FxiOS/1.0 Mobile/12F69 Safari/600.1.4").get();
System.out.println(doc.html());
} catch(IOException e) {
e.printStackTrace();
}
}
}
I have to display a webpage inside my Java software.( you can think it is one the page of a Wireless Device configuration)
But when Im gonna go to that page in browser,browser displays a user and password popup and I enter the user and password and go to that page.( notice that username is always root, but password can be different)
Now I am gonna display the page from Java software, I could link the page and open the page via Java, But the host inside the wireless device displays :
401 Unauthorized, The URL of the page I want to display is http://192.168.1.2/Wireless
I used Fiddler to monitor its behavior, that URL is continuously reloading.
And here is the Header of the Get method of that :
GET /Wireless HTTP/1.1
Host: 192.168.1.2
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.1.2/Wireless
Authorization: Basic xxxxxxxxx
Connection: keep-alive
How can I do that? With HttpClient ?
Does anybody knows how to do that? And give me a solution?
Or a sample or anything can help me?|
Thanks.
EDIT :
Here is my code to display the webview :
myFrame.setSize(mainJTabbed.getSize());
myFrame.setLocationRelativeTo(mainJTabbed);
myFrame.setVisible(true);
myFrame.add(myFXPanel);
Platform.runLater(() -> {
BorderPane borderPane = new BorderPane();
WebView webComponent = new WebView();
webComponent.getEngine().load("myURL");
borderPane.setCenter(webComponent);
Scene scene = new Scene(borderPane,450,450);
myFXPanel.setScene(scene);
});
And even it is a GET method, the user and password does not post via URL.
I did it in this way, I created a class with a constructor, the all is in constructor, notice that my constructor what I wanted, you can change it, but the algorithm is this :
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.InetAddress;
import java.net.MalformedURLException;
import java.net.PasswordAuthentication;
import java.net.URL;
import javafx.application.Platform;
import javafx.embed.swing.JFXPanel;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
import javafx.scene.web.WebView;
import javax.swing.JFrame;
public class WebViewSample {
public WebViewSample(String urlString, JFrame myFrame, JFXPanel myFXPanel) {
try {
// Sets the authenticator that will be used by the networking code
// when a proxy or an HTTP server asks for authentication.
Authenticator.setDefault(new CustomAuthenticator());
URL url = new URL("http://" +urlString + "/wireless");
Platform.runLater(() -> {
BorderPane borderPane = new BorderPane();
WebView webComponent = new WebView();
webComponent.getEngine().load(url.toString());
borderPane.setCenter(webComponent);
Scene scene = new Scene(borderPane,450,450);
myFXPanel.setScene(scene);
});
}
catch (MalformedURLException e) {
System.out.println("Malformed URL: " + e.getMessage());
}
catch (IOException e) {
System.out.println("I/O Error: " + e.getMessage());
}
}
public static class CustomAuthenticator extends Authenticator {
// Called when password authorization is needed
protected PasswordAuthentication getPasswordAuthentication() {
// Get information about the request
String prompt = getRequestingPrompt();
String hostname = getRequestingHost();
InetAddress ipaddr = getRequestingSite();
int port = getRequestingPort();
String username = "myUserName";
String password = "myPassword";
// Return the information (a data holder that is used by Authenticator)
return new PasswordAuthentication(username, password.toCharArray());
}
}
}
I am trying to upload file to a url for that I am using this code
import java.io.File;
import java.io.IOException;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.multipart.FilePart;
import org.apache.commons.httpclient.methods.multipart.MultipartRequestEntity;
import org.apache.commons.httpclient.methods.multipart.Part;
public class UploadIt{
public static void main(String[] args) throws IOException {
String s=uploadFile(new File("C://paid.png"), "http://abc.xyz.com");
System.out.println("val is "+s);
}
public static String uploadFile(File resourceUrl,String url) throws HttpException, IOException{
File f = resourceUrl;
PostMethod filePost = new PostMethod(url);
Part[] parts = {new FilePart(f.getName(), f)};
filePost.setRequestEntity(new MultipartRequestEntity(parts, filePost.getParams()));
HttpClient client = new HttpClient();
int status = client.executeMethod(filePost);
String resultUUid=null;
resultUUid = filePost.getResponseBodyAsString();
filePost.releaseConnection();
System.out.println(" status "+status );
return resultUUid;
}
}
From source.
It is giving error
status 406
val is <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>406 Not Acceptable</title>
</head><body>
<h1>Not Acceptable</h1>
<p>An appropriate representation of the requested resource / could not be found on this server.</p>
<p>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
How to resolve this problem
my directory has permission 755
It's not the directory permissions. it may be a restriction on the Mime types accepted by the server - have a look here http://www.checkupdown.com/status/E406.html
Accept: The MIME types accepted by the client. For example, a browser may only accept back types of data (HTML files, GIF files etc.) it knows how to process.
maybe you could print the response headers as well for further debugging
I am trying to downloada picture from a certain url, but cant do so because I somehow have to give the right userclient to the website.I am sure the problem is that I cant give the user client while using the Url class, because the page can be accesed via browser. I tried using proxy and Urlconnection but couldnt get it to work. Please share your toughts on the matter!
My code is the following:
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.SocketAddress;
import java.net.URL;
import java.net.URLConnection;
import javax.imageio.ImageIO;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.highgui.Highgui;
public class KepLetolto {
public static void main(String[] args) throws IOException {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
File file = new File("tempjpeg");
SocketAddress address = new java.net.InetSocketAddress("xyz.com", 8080);
// Create an HTTP Proxy using the above SocketAddress.
Proxy proxy = new Proxy(Proxy.Type.HTTP, address);
URL url_kep =new URL("http://www.theouthousers.com/images/templates/thumbnails/128058/bayfinger_size3.png");
ImageIO.write(ImageIO.read(url_kep), "jpeg", file);
Mat uj = Highgui.imread("temp.jpeg" ,Highgui.CV_LOAD_IMAGE_COLOR);
}
}
Instead of using ImageIO.read(URL), which limits you to the default behavior of the URL's underlying URLConnection, use ImageIO.read(InputStream).
This allows you to use any HTTP client library - including the basic HttpURLConnection, which you can get from (HttpURLConnection)url_kep.openConnection(). Using that, you can set headers such as User-Agent, if that's the header required by the site, or other headers such as Referer which are sometimes used to prevent deep-linking.
Once you set up all the headers and any other request options, you can get an InputStream from the client object, and pass that to ImageIO.
This Solution Worked For Me:
URLConnection openConnection = new URL("YOUR_IMAGE_URL").openConnection();
openConnection.addRequestProperty("User-Agent", "YOUR USER AGENT");
InputStream is = openConnection.getInputStream();
BufferedImage saveImage = ImageIO.read(is);
ImageIO.write(saveImage, "png", new File("PATH\\TO\\IMAGE\\FILE.PNG"));
I am trying to come up with a Android app that needs some information on the university inner website. I have been trying to use Jsoup to login the website programmatically. Here is the code I have now:
import org.jsoup.Connection;
import org.jsoup.Connection.Method;
import org.jsoup.Jsoup;
//import org.jsoup.helper.Validate;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
//import org.jsoup.select.Elements;
import java.io.IOException;
import java.util.Map;
public class Test {
public static void main(String[] args) {
Document doc;
try {
Connection.Response res = Jsoup
.connect(
"https://sso.bris.ac.uk/sso/login?service=https%3A%2F%2Fwww.cs.bris.ac.uk%2FTeaching%2Fsecure%2Funit-list.jsp%3Flist%3Dmine")
.execute();
Map<String, String> cookies = res.cookies();
System.out.println(cookies.keySet());
Document fakepage = res.parse();
Element fakelt = fakepage.select("input[name=lt]").get(0);
Element fakeexecution = fakepage.select("input[name=execution]")
.get(0);
Element fake_eventID = fakepage.select("input[name=_eventId]").get(
0);
System.out.println("Hello World!");
System.out.println(fakelt.attr("value"));
System.out.println(fakeexecution.toString());
System.out.println(fake_eventID.toString());
// System.out.println(cookies.get("JSESSIONID"));
String url="https://sso.bris.ac.uk/sso/login?service=https%3A%2F%2Fwww.cs.bris.ac.uk%2FTeaching%2Fsecure%2Funit-list.jsp%3Flist%3Dmine";
System.out.println(url);
Connection newreq = Jsoup
.connect(url)
.cookies(cookies).data("lt", fakelt.attr("value")).followRedirects(true).header("Connection", "keep-alive")
.header("Refer", " https://sso.bris.ac.uk/sso/login?service=https%3A%2F%2Fwww.cs.bris.ac.uk%2FTeaching%2Fsecure%2Funit-list.jsp%3Flist%3Dmine")
.header("Content-Type","application/x-www-form-urlencoded;charset=UTF-8")
.userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:27.0) Gecko/20100101 Firefox/27.0")
.data("lt",fakelt.attr("value"))
.data("execution", fakeexecution.attr("value"))
.data("_eventID", fake_eventID.attr("value"))
.data("username", "aabbcc").data("password", "ddeeff")
.data("submit", "").method(Method.POST);
Connection.Response newres = newreq.execute();
doc = newres.parse();
System.out.println(doc.toString());
System.out.println(newres.statusCode());
Map<String,String> newcookies = newres.cookies();
doc = Jsoup.connect("https://www.cs.bris.ac.uk/Teaching/secure/unit-list.jsp?list=mine").cookies(newcookies).get();
System.out.println(doc.toString());
// System.out.println(doc.toString());
} catch (IOException e) {
System.out.println("Excepiton:");
System.out.println(e.getMessage());
}
}
}
I completely faked a form to submit use Jsoup, and to get around the security cookies I first request the website once and then use the cookies it sent me to request the website again. The form has some hidden fields so I use the ones I got on my first request to fake it when I request it again. However this does not work. Is it possible to do it or the server has some advanced preventer against me doing so?
Do not use Jsoup to do this, it needs you to handle all the cookies yourself, instead, use Httpclient, if you use something from 4.0 onward it handle the cookies automatically. Much eaiser to work with.