We Keep Coding

An issue with an URLConnection using java

I'm trying to read out the code of a website.
But there is an issue if I want to receive the code of this site for example: "https://www.amazon.de/gp/bestsellers/pet-supplies/#2"
I tried a lot, but still im just receiving the code of https://www.amazon.de/gp/bestsellers/pet-supplies". So something does not work right as I want to receive place 21-40 and not 1-20.
I'm using an URLConneciton and a BufferedReader:
public String fetchPage(String urlS){
String s = null;
String qc = null;
try{
URL url = new URL(urlS);
URLConnection uc = url.openConnection();
uc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0");
BufferedReader reader = new BufferedReader(new InputStreamReader(uc.getInputStream()));
while((s = reader.readLine()) != null){
qc += s;
}
reader.close();
} catch(IOException e) {
e.printStackTrace();
qc = "receiving qc failed";
}
return qc;
}
Thank you in advance for your effort :)

The URL you're fetching, contains an achor (the #2 at the end). An anchor is a client-side concept and is originally used to jump to a certain part of the page. Some webapps (mostly single-page apps) use the anchor to keep track of some sort of state (eg. what page of products you're viewing).
Since the anchor is a client side concept, the responding webserver (or your browser/HTTP client library) just drops any anchors as if you actually requested https://www.amazon.de/gp/bestsellers/pet-supplies.
Bottom line is that you'll never get the second page... Goog luck in scraping Amazon though ;)

Get Metadata from Dropbox Link Without Auth

I want to check for a version changed/get metadata of a text-file with a shared link on dropbox. I will not be using dropbox api as it makes users use their own accounts. I want them to link to my account and I cannot do that manually since I might change my password later.
so: no auth token, just get metadata from shared link of dropbox so that I can check for version changes and if the version has changed download the contents of the new file.
also: I'm open to other suggestions to make this work as well. Please explain in a little detail your solution.
Updated E-Tag Issue:
public void getFromOnlineTxtDatabase(){
try{
URL url = new URL("url-here");
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setDoOutput(false);
con.setReadTimeout(20000);
con.setRequestProperty("Connection", "keep-alive");
//get etag for update check
String etag = con.getHeaderField("etag");
//String etag= "";
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0");
((HttpURLConnection) con).setRequestMethod("GET");
//System.out.println(con.getContentLength()) ;
con.setConnectTimeout(5000);
BufferedInputStream in = new BufferedInputStream(con.getInputStream());
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
System.out.println(responseCode);
}
StringBuffer buffer = new StringBuffer();
int chars_read;
//int total = 0;
while ((chars_read = in.read()) != -1)
{
char g = (char) chars_read;
buffer.append(g);
}
final String page = buffer.toString();
//create password_ems.txt to internal
if (fileExistance("data.txt")){
File dir = getFilesDir();
File file = new File(dir, "data.txt");
boolean deleted = file.delete();
stringToTxt(page, "data.txt");
}else{
stringToTxt(page, "data.txt");
}
if (fileExistance("data_etag.txt")){
File dir = getFilesDir();
File file = new File(dir, "etag.txt");
boolean deleted = file.delete();
stringToTxt(etag, "etag.txt");
}else{
//create etag_file
stringToTxt(etag, "data_etag.txt");
}
// Log.i("Page", page);
}catch(Exception e){
showDialog("Database Fetch Failure","Unable to Fetch Password Database, check your internet" +
" connection and try again later.",0);
Log.i("Page", "Error");
}
}

If you do an HTTP HEAD request against a public or shared Dropbox URL, you'll get, among other things, an etag header. I don't know that this behavior is guaranteed, since I don't think it's documented anywhere, but at least for now the etag header can be used to determine when a file has changed. (If the etag is different, the file has changed.)
EDIT
In general when using ETags, the most efficient thing to do is issue a GET request with a header of If-None-Match: <old etag>. If the content hasn't changed, this will respond with a 304, but if the content has changed, this will download the new content as per a normal GET request (and the response will be 200).

how to NOT get mobile version of a website in java?

This code returns the mobile version of a website. How to get a desktop version instead?
InputStreamReader page = new InputStreamReader(new URL('http://www.***.com/').openStream());

Use a user agent that matchs a Desktop browser
Edited with sources
URL url = new URL("http://www.clarku.edu/");
URLConnection connection = url.openConnection();
connection.addRequestProperty("User-Agent", "Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1");
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));

HttpURLConnection to Tomcat

I am trying to connect from a java desktop application to a jsp Servlet to send a file.
Clientcoding:
HttpURLConnection urlConnection = null;
URL url = null;
url = new URL("http://127.0.0.1:8080/emobile/AddTripMobile");
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setDoOutput(true);
OutputStream out = new BufferedOutputStream(
urlConnection.getOutputStream());
out.write(12); //The data to send
out.flush();
If I connect with the desktop application to the server nothing happens.
(I set a breakpoint in the doGet and doPost)
Any suggestions?

You need to add the following :
InputStream is = urlConnection.getInputStream();
out.write(12); //The data to send
out.flush();

Try closing the output stream.

Need Java Expert Help on Executing HTTP GET with Cookie

My problem is that I want to use Java to implement an application which sends an HTTP GET request to some website. However, the target website needs one cookie to be set:
ShippingCountry=US
If this cookie is not set it returns bad indications. Below are my code segment and I get null from connect().
String urlString = "http://www1.macys.com/catalog/index.ognc?CategoryID=5449&viewall=true";
try{
URL url = new URL(urlString);
URLConnection connection = url.openConnection();
connection.addRequestProperty("Cookie", "ShippingCountry=US");
connection.connect();
// Create file
FileWriter fstream = new FileWriter("d:/out.txt");
BufferedWriter out = new BufferedWriter(fstream);
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;
while ((line = rd.readLine()) != null)
{
out.write(line);
}
rd.close();
//Close the output stream
out.close();
}
Can somebody help me?

Just a guess but perhaps you might need setRequestProperty instead of addRequestProperty, since there can be only one Cookie string in a request.
connection.setRequestProperty("Cookie", "ShippingCountry=US");
If you need to send multiple cookie values you tack them on with colons:
connection.setRequestProperty("Cookie", "ShippingCountry=US;OtherValue=2");
Update:
I tried doing a GET request using python and it looks like I get a 500 error. Perhaps the server is blocking the request because it doesn't look like a web browser. Maybe adding the following headers will work:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive
In particular the Accept and User-Agent headers might be required.