i'm starting with Programming Java and wan't to create a simple "Backend Crawler". For this i need a login function by Post thats not problem but how i can do the cookies wars saved and on the next request the Script don't need login again?
Can you give me a example? I can't find a solution in the internet.
Maybe you can explain me how i can do the next request with the Cookies from the first Page? :)
Hope for your answer.
Sorry for my bad english.
Here is my first Logintest :P
import java.util.ArrayList;
import java.util.List;
import org.apache.http.HttpEntity;
import org.apache.http.NameValuePair;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class Login {
public static String loginAndGetHTML() throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
String html;
HttpPost HttpPost = new HttpPost("http://www.google.com");
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("username", "admin"));
nvps.add(new BasicNameValuePair("password", "1234"));
HttpPost.setEntity(new UrlEncodedFormEntity(nvps));
HttpPost.addHeader("Referer", "http://tutorials.amazingcode.de/login/index.php");
HttpPost.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0");
CloseableHttpResponse response = httpclient.execute(HttpPost);
try {
HttpEntity entity = response.getEntity();
html = EntityUtils.toString(entity);
EntityUtils.consume(entity);
} finally {
response.close();
}
if(html.contains("Falsche Nutzerdaten")) {
throw new Exception("Login fehlgeschlagen");
}
return html;
}
public static String parseHTML(String html) throws Exception {
Document doc = Jsoup.parse(html);
String zahl = doc.getElementById("zahl").text();
return zahl;
}
}
you need to retrieve the Set-Cookie header from the response, store its value,
and ask following requests with Cookie header and that value.
see this post for more explainer on cookies How are cookies passed in the HTTP protocol?
Related
I'm using Apache HttpClient version 4.5.13 and I'm having trouble creating my POST request body. From the tutorials I've seen online, they tell me to use NameValuePair when creating the request entity. However, NameValuePair only accepts String for the values.
How can I set Integer, Double, and Booleans as well? The API I'm calling has a mixture of them in the JSON body. For example, my body can look like this:
{
"id": 3,
"name": "Test",
"accepted": false,
"street": "foo bar ave."
}
This is how the documentations recommend I create the body:
import org.apache.http.NameValuePair;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
public static void main(String[] args) {
List<NameValuePair> body = new ArrayList<NameValuePair>();
body.add(new BasicNameValuePair("id", Integer.valueOf(3)));
body.add(new BasicNameValuePair("name", "Test"));
body.add(new BasicNameValuePair("accepted", Boolean.valueOf(false)));
body.add(new BasicNameValuePair("street", "foo bar ave."));
HttpPost httpPost = new HttpPost("http://my-url.com/test-api");
httpPost.setEntity(new UrlEncodedFormEntity(body));
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-Type", "application/json");
CloseableHttpClient httpClient = HttpClients.createDefault();
CloseableHttpResponse response = httpClient.execute(httpPost);
// etc...
}
But as you can see, I have to convert everything to String which the API I'm calling rejects because of the mismatch of data types. Is there some other way I can create the request which accepts various data types?
I was able to fix this by using StringEntity. I converted the code in the question to the following:
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.json.JSONObject;
public static void main(String[] args) {
JSONObject jsonObject = new JSONObject();
jsonObject.put("id", 3);
jsonObject.put("name", "Test");
jsonObject.put("accepted", false);
jsonObject.put("street", "foo bar ave.");
HttpPost httpPost = new HttpPost("http://my-url.com/test-api");
httpPost.setEntity(new StringEntity(jsonObject.toString()));
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-Type", "application/json");
CloseableHttpClient httpClient = HttpClients.createDefault();
CloseableHttpResponse response = httpClient.execute(httpPost);
// etc...
}
And I was able to call the API in question without any errors. Hopefully, this helps someone!
When making a post request to my backend, I get the following error. I can't really find a fix online and am very new to Java.
Here is the error:
Caused by: java.lang.ClassNotFoundException: org.apache.http.concurrent.Cancellable
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 59 more
Code
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.http.HttpHeaders;
import java.util.ArrayList;
import java.util.List;
public class APIBridge {
private final CloseableHttpClient httpClient = HttpClients.createDefault();
public String url = "http://localhost:9004/index.php";
public static void Register(String username, String password) throws UnsupportedEncodingException {
String url = "http://localhost:9004/index.php";
HttpPost post = new HttpPost(url);
// Add request parameter, form parameters
List<NameValuePair> urlParameters = new ArrayList<>();
urlParameters.add(new BasicNameValuePair("username", username));
urlParameters.add(new BasicNameValuePair("password", password));
urlParameters.add(new BasicNameValuePair("Register", "true"));
post.setEntity(new UrlEncodedFormEntity(urlParameters));
try (CloseableHttpClient httpClient = HttpClients.createDefault();
CloseableHttpResponse response = httpClient.execute(post)) {
System.out.println(EntityUtils.toString(response.getEntity()));
} catch (ClientProtocolException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private void close() throws IOException {
httpClient.close();
}
private void sendGet() throws Exception {
HttpGet request = new HttpGet("https://www.google.com/search?q=mkyong");
// Add request headers
request.addHeader("custom-key", "mkyong");
//request.addHeader(, "Googlebot");
try (CloseableHttpResponse response = httpClient.execute(request)) {
// Get HttpResponse Status
System.out.println(response.getStatusLine().toString());
HttpEntity entity = response.getEntity();
Header headers = entity.getContentType();
System.out.println(headers);
if (entity != null) {
// Return it as a String
String result = EntityUtils.toString(entity);
System.out.println(result);
}
}
}
IMO, it is easier to use the HTTP client in the JDK, so I have provided an answer demonstrating this alternate approach.
The Apache HTTP libraries were never targeted at somebody who is "new to Java". If you use them, make sure you use the correct versions of all the libraries and matching documentation as the library has gone through multiple incompatible API changes over years of library upgrades.
From your question, your module-info.java may be wrong or may need to be deleted. But your issue could also be caused by an environment setup or dependency version issue.
Basically, the class isn’t on the class or module path. Or it is on a path, but it is not accessible. The reason for that is not discernable from your question.
Example JavaFXApp using JDK HTTP Client
Makes a post request, gets the result as text and places the text in a TextArea.
module-info.java
module com.example.httpapp {
requires javafx.controls;
requires java.net.http;
exports com.example.httpapp;
}
src/main/java/com/example/httpapp/HttpApp.java
package com.example.httpapp;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.TextArea;
import javafx.stage.Stage;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
public class HttpApp extends Application {
#Override
public void start(Stage stage) throws IOException, URISyntaxException, InterruptedException {
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("http://jsonplaceholder.typicode.com/posts"))
.POST(HttpRequest.BodyPublishers.ofString("Sample Post Request"))
.build();
HttpResponse<String> response = client.send(
request,
HttpResponse.BodyHandlers.ofString()
);
String responseBody = response.body();
TextArea textArea = new TextArea(responseBody);
textArea.setStyle("-fx-font-family: monospace;");
stage.setScene(new Scene(textArea));
stage.show();
}
public static void main(String[] args) {
launch();
}
}
Authentication
Outside of a local test app, you should not send unencrypted authentication data over a network connection as you have in your question code.
If you also need authentication, it is best to communicate over HTTPS. Then you can securely use basic or digest authentication, or HTTPS mutual authentication, or encode the authentication information in the post body.
Examples for basic authentication using the Java HTTP client are at:
Baeldung: Java HttpClient Basic Authentication
I have been able to successfully authentication to a service that requires ntlm authentication when using the WinHttpClients and a GET request. However when I try to do a POST I always get a 401 return code. Has anyone done this sucessfully before?
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.NTCredentials;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.WinHttpClients;
public class WindowsAuthPOst {
public static void main (String []args) throws Exception, IOException
{
org.apache.log4j.BasicConfigurator.configure();
CloseableHttpClient httpclient = WinHttpClients.createDefault();
HttpHost target = new HttpHost("SomeHost.domain", 443, "https");
HttpClientContext context = HttpClientContext.create();
HttpGet httpget = new HttpGet("/some/Service.svc");
CloseableHttpResponse response1 = httpclient.execute(target, httpget, context);
try {
HttpEntity entity1 = response1.getEntity();
} finally {
response1.close();
}
// Execute an expensive method next reusing the same context (and connection)
HttpPost httppost = new HttpPost("/some/Service.svc");
httppost.setHeader("SOAPAction", "Some Soap Action");
httppost.setEntity(new StringEntity("Soap Payload"));
CloseableHttpResponse response2 = httpclient.execute(target, httppost, context);
try {
HttpEntity entity2 = response2.getEntity();
} finally {
response2.close();
}
}
}
You can check if it is available with.
if (!WinHttpClients.isWinAuthAvailable()) {
System.out.println("Integrated Win auth is not supported!!!");
}
If not, it could be that you do not have jna.jar in your classpath. It depends on jna and will silently return false on the above if it not there, see source code.
Try with get (or options) before post. Some webservers requires that because of CORS.
https://stackoverflow.com/a/38410411/2376661
I'm having a problem: I've just copied the following code from NetBeans to Eclipse (an ADT project). I've imported all the same librarys I used in NetBeans, but I have 2 errors, in the following lines:
EntityUtils.consume(entity); - The method consume(HttpEntity) is undefined for the type EntityUtils
httpPut.releaseConnection(); - The method releaseConnection() is undefined for the type HttpPut
Complete code:
import java.util.ArrayList;
import java.util.List;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.util.EntityUtils;
import com.android.lul.classes.User;
public class UserService {
private static final String BASE_URI = "http://localhost:8080/LULServices/webresources";
public static String Login (String login, String password, String ipAdd)
{
String toReturn = null;
final DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("localhost", 8080),
new UsernamePasswordCredentials("xxxx", "xxxx"));
HttpPut httpPut = new HttpPut(BASE_URI + "/services.users/login");
HttpConnectionParams.setConnectionTimeout(httpclient.getParams(), 10000);
httpPut.addHeader("Content-type", "multipart/form-data");
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("login", "login"));
nameValuePairs.add(new BasicNameValuePair("password", "password"));
httpPut.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = httpclient.execute(httpPut);
try {
HttpEntity entity = response.getEntity();
String putResponse = EntityUtils.toString(entity);
toReturn = putResponse;
EntityUtils.consume(entity);
} finally {
httpPut.releaseConnection();
}
} finally {
httpclient.getConnectionManager().shutdown();
return toReturn;
}
Can you help me?
Thanks
Android comes with a prepackaged version of Apache HttpClient that doesn't have those methods. They are no longer supporting development and that code is outdated.
The Android team recommends that you use HttpUrlConnection for new code instead of HttpClient. More information can be found at this blog on the Android Developers site.
I think you should follow a simpler example, as this one seems to be meant to upload files (multipart/form-data), but regarding the specific problems in your code:
Instead of EntityUtils.consume(entity); you can do entity.consumeContent();
http://developer.android.com/reference/org/apache/http/HttpEntity.html
The HTTPRequest classes (HttpPost, HttpPut) don't have any releaseConnection() method. Probably because they don't need to be released (somebody correct me if I'm wrong). That method is usually applicable for persistent connections which is not the case here.
Check your imports; What version of Apache HttpComponents are you using in your new project?
How to make HTTPClient use custom User-Agent header?
The following code submits empty user-agent. What am I missing?
import java.io.IOException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.CoreProtocolPNames;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
public class TestHTTP {
public static void main(String[] args) throws ClientProtocolException, IOException {
HttpGet request = new HttpGet("http://tool.keepmeapi.com/echo");
HttpContext HTTP_CONTEXT = new BasicHttpContext();
HTTP_CONTEXT.setAttribute(CoreProtocolPNames.USER_AGENT, "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13");
request.setHeader("Referer", "http://www.google.com");
HttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(request, HTTP_CONTEXT);
if (response.getStatusLine().getStatusCode() < 200 || response.getStatusLine().getStatusCode() >= 400) {
throw new IOException("Got bad response, error code = " + response.getStatusLine().getStatusCode());
}
HttpEntity entity = response.getEntity();
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
EntityUtils.consume(entity);
}
}
}
Note: The solution is for users using the old httpcomponents 4.2 and before.
The line
request.setHeader("User-Agent", "MySuperUserAgent");
is missing. Add it and enjoy.
Note: Starting from httpcomponents 4.3 this solution is deprecated.
You can also set a global user agent value instead of per request:
String userAgent = "NewUseAgent/1.0";
HttpClient httpClient = new DefaultHttpClient();
httpclient.getParams().setParameter(CoreProtocolPNames.USER_AGENT, userAgent);
With httpcomponents 4.3 you should use the client builder to set the user agent:
HttpClient httpClient = HttpClients.custom()
.setUserAgent("my UserAgent 5.0")
.build();
httpClient.execute(new HttpGet("http://www.google.de"));