I'm sending some string eg:
private final String test = "{\"data\":{\"type\":\"test\",\"attributes\":{\"color\":\"yellow\",\"name\":\"TestN\"}}}";
via Rest Assured
given()
.header("Origin", "http://localhost:5000")
.header("Accept-Encoding", "gzip, deflate, br")
.header("Accept-Language", "pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7")
.header("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36")
.header("Content-Type", "application/vnd.api+json")
.header("Accept", "application/vnd.api+json")
.header("Cookie", "xxxxxx")
.header("Connection", "keep-alive")
.header("Cache-Control", "no-cache")
.header("Host", "localhost:4400")
.body(test).with()
.log().everything()
.when()
.post(base + "test-endpoint")
.then().statusCode(201);
unfortunately API responds with 500. I'm sending identical request via Postman and it works perfectly. Only difference is "assings" section. After Postman request it looks like:
assigns: %{
doc: %Jabbax.Document{
data: %Jabbax.Document.Resource{
attributes: %{"appointment_color" => "yellow", "name" => "TestN"},
id: nil,
links: %{},
meta: %{},
relationships: %{},
type: "test"
},
errors: [],
included: [],
jsonapi: %{version: "1.0"},
links: %{},
meta: %{}
}
},
when after Rest Assured request it's empty:
assigns: %{},
all of the headers are added and I've tried sending it as a string parsed from .json file. Everything gives same results. Somebody know what can be the problem?
The clue was that REST Assured added a charset information - similar issue was described here.
Related
I am trying to parse through a particular website and am getting an HTTP response code of 419 when my java code calls it. I need to parse through the response to find content and I am stuck on the response code.
I have tried putting together a Java program using apache http client(version 4.5.6) to call a website that I need to parse. The http response code I get back is 419.
try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
HttpGet httpGet = new HttpGet("http://www.website.com");
try (CloseableHttpResponse response1 = httpclient.execute(httpGet)) {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
EntityUtils.consume(entity1);
}
}
The result that it prints out is this:
HTTP/1.1 419 status code 419
I am expecting a 200
HTTP/1.1 200 OK
I get that when I change the website to google or other sites.
I was making a get request through HttpClient library as well as from POSTMAN and facing same 419 error. To solve this 419 error we need to add csrf token while making form submission.
However, In-case if you are still wondering how to find csrf token even when you are making a GET request and facing status 419. In my case I solved the problem by adding the user-agent: xxxx token in header.
Example:
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
HttpClient Code:
connectionManager = new PoolingHttpClientConnectionManager();
...
...
...
httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setRedirectStrategy(new LaxRedirectStrategy())
.setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36")
.build();
I'm trying to rebuild the session setup to a web server by a Java HttpClient application. I have chosen the incubated HttpClient provided with Java 9 and Java 10.
With Chrome I captured this headers from a single request:
General
Request URL: https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0
Request Method: GET
Status Code: 302 Found
Remote Address: <theProxy>:8000
Referrer Policy: no-referrer-when-downgrade
Response Headers
Connection: Keep-Alive
Content-Length: 164
Content-Type: text/html; charset=iso-8859-1
Date: Fri, 08 Jun 2018 14:33:16 GMT
Keep-Alive: timeout=300, max=100
Location: https://<another_url>:443/nesp/app/plogin?agAppNa=app_me_company_ext&c=secure/name/password/uri&target=%22https://<another-usr>/browseIssues.spr?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0%22
P3p: CP="NOI"
Server: Apache
Set-Cookie: IPCZQX03224bfb75=030003000000000000000000000000008f7aed69; path=/; domain=.me.de
Via: 1.1 <host> (Access Gateway-ag-7169149846802036-13837511)
Request Headers
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Cookie: org.ditchnet.jsp.tabs.wiki=wiki-wysiwyg; ZNPCQ003-31393000=6c2f99a3; ZNPCQ003-32323200=cd188fdd
DNT: 1
Host: <host>
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
Query String Parameters
user_id: 1176
onlyDirectUserItems: true
onlyAssignedToUser: true
show: Unresolved
itemsFilter: 0
What can be seen the Response Header provides a URL (header-key: "location") which I need to grab and call next. But with my http client I fail with status-code 400 and get almost nothing
This is my code
url = "https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0";
HttpClient client = HttpClient.newBuilder()
.proxy(ProxySelector.of(new InetSocketAddress("<theProxy>", 8000)))
.cookieHandler(new CookieManager(null, CookiePolicy.ACCEPT_ALL))
.followRedirects(HttpClient.Redirect.SAME_PROTOCOL)
.build();
HttpRequest request = HttpRequest.newBuilder()
.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0")
.header("Upgrade-Insecure-Requests", "1")
// .header("Host", "<host>")
.header("Connection", "keep-alive")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8")
.header("Accept-Language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7")
.header("Accept-Encoding", "gzip, deflate, br")
.uri(new URI(url))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandler.asString());
HttpHeaders headers = response.headers();
Map<String, List<String>> headerMap= headers.map();
for (String key : headerMap.keySet()) {
System.out.println(">"+key+"<");
for (String value : headerMap.get(key)) {
System.out.println(" " + value);
}
}
System.out.println(response.statusCode());
System.out.println(response.body());
I have no clue what might be wrong and how to proceed to get this done. I hope someone can tell me what to ty next.
What I also do not understand: I had to remove the header "Host" - because I got the response: "Your browser sent a request that this server could not understand."
The very same header as can be found in the Chrome-listing
I could get it run with Apache HttpClient. I cannot tell what is wrong with the incubated HttpClient - but there is for sure a reason that it is not a fully integrated part of the Java 9/10 delivery
I'm trying to get the most recents comments from youtube videos, i'm replicating the POST petition as i see from the Firebug plugin.
The response code i get after executing the code is the 403
The 403 or "Forbidden" error message is a HTTP standard response code
indicating that the request was legal and understood but the server
refuses to respond to the request.
I'm not quite sure what info i'm missing to successfully get the comments from the video.
Map<String, String> cookies =new HashMap<String,String>();
cookies.put("session_token", session_token);
cookies.put("page_token", page_token);
Response res= Jsoup.connect(url_comments).header("Accept-Encoding", "gzip, deflate")
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0")
.referrer(linkVideo)
.data(cookies)
.header("Host", "www.youtube.com")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.header("Accept-Language", "es-MX,es-ES;q=0.9,es;q=0.7,es-AR;q=0.6,es-CL;q=0.4,en-US;q=0.3,en;q=0.1")
.header("Accept-Encoding", "gzip, deflate, br")
.header("X-YouTube-Client-Name", "1")
.header("X-YouTube-Client-Version", "1.20161213")
.header("X-YouTube-Page-CL", "141992279")
.header("X-YouTube-Page-Label", "youtube_20161213_0_RC1")
.header("X-YouTube-Variants-Checksum", "a55ebd64c41b17f87231bfa795156a50")
.header("Content-Type", "application/x-www-form-urlencoded")
.header("Cookie", "VISITOR_INFO1_LIVE=TRli2T2GQEQ; PREF=cvdm=list&gl=DE&f1=50000000&f5=30; _ga=GA1.2.527380759.1481667156; CONSENT=WP.25af40; YSC=vwoNBo1vTsg; ST-os04us=itct=CAIQ7pgBIhMI1Lef15X30AIVg2ROCh2zVgAw&csn=Bg1TWLXdKoSGuwXKyYzADg")
.header("Connection", "keep-alive").ignoreContentType(true)
.method(Method.POST).execute();
the session_token and page_token is info i get directly from the video webpage. Need help to solve this and get the first comments with the POST petition.
I am able to curl this url to get the access token for Spotify Web API using the below call from command line:
curl -H "Authorization: Basic <Base64 client_id:client_secret>" -d grant_type=client_credentials https://accounts.spotify.com/api/token
And I get a response with the api_token. But when I try to do this in java through Jsoup, I get a 405 error:
Map<String, String> data = Maps.newHashMap();
data.put("grant_type", "client_credentials");
String clientCred =
new String(
Base64.encodeBase64((CLIENT_ID + ":" + CLIENT_SECRET).getBytes()));
String url = "https://accounts.spotify.com/api/token";
Document doc = Jsoup.connect(URIUtil.encodeQuery(url))
.header("Accept-Language", "en")
.header("Authorization", "Basic " + clientCred)
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.data(data)
.ignoreHttpErrors(true)
.header("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9"
+ ",*/*;q=0.8")
.get();
Am I doing something wrong in setting the request parameters? I tried a simple HttpURLConnection too, with the same request parameters, and that also fails with the same 405 error.
When you use -d flag in curl you are actually sending a POST. If you make a POST with Jsoup It will work.
But Jsoup will complain with a org.jsoup.UnsupportedMimeTypeException because server will respond with Content-Type: application/json. Then you have to add an .ignoreContentType(true) and you are done.
The code would be:
Document doc = Jsoup.connect(url)
.header("Accept-Language", "en")
.header("Authorization", "Basic " + clientCred)
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.data(data)
.ignoreHttpErrors(true)
.ignoreContentType(true)
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9"+ ",*/*;q=0.8")
.post();
Bytheway, I'm getting a {"error":"invalid_client"} because I don't have a key, but I think you'll be fine.
I hope it will help.
I have set up a Apache server, with a php script running to handle various post requests. the script looks as followed:
$uploaddir = realpath('mypath');
//realpath('./') . '/';
$uploadfile = $uploaddir . '/' . basename($_FILES['file_contents']['name']);
echo '<pre>';
if (move_uploaded_file($_FILES['file_contents']['tmp_name'], $uploadfile)) {
echo "File is valid, and was successfully uploaded.\n";
} else {
echo "Possible file upload attack!\n";
}
echo 'Here is some more debugging info:';
print_r($_FILES);
echo "\n<hr />\n";
print_r($_POST);
print "</pr" . "e>\n";
and this works great, it handles the request and moves my file to where i want it.
But when i try this with following java client, i dont get anything:
String URL = "http://localhost/accept.php";
File file = new File("C:/Ozer/ANPRProject/MDT/Export/test.txt");
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(URL);
try{
System.out.println(""+file.exists());
InputStreamEntity reqEntity = new InputStreamEntity(new FileInputStream(file),-1);
reqEntity.setContentType("application/octet-stream");
reqEntity.setChunked(true);
httppost.setEntity(reqEntity);
System.out.println("execute request " + httppost.getRequestLine());
HttpResponse response = httpclient.execute(httppost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
System.out.println("Chunked: " + resEntity.isChunked());
}
EntityUtils.consume(resEntity);
} catch(Exception e) {
} finally {
httpclient.getConnectionManager().shutdown();
}
}
What I get as system out put is:
true
execute request POST http://localhost/accept.php HTTP/1.1
----------------------------------------
HTTP/1.1 200 OK
Response content length: 330
Chunked: false
Which I find weird, because the chunked has not been set to true? While i specifically asked for it in my code. But anyhow, the uploading does not work. I Have set the contentType to "application/octet-stream" because the php script I used to request a post gave me as feedback: [type] => application/octet-stream
So the logs of the apache show me that the connection was successful and he indeed did get the request:
127.0.0.1 - - [08/Aug/2013:16:10:04 +0200] "POST /accept.php HTTP/1.1" 200 330 "-" ""Apache-HttpClient/4.2.5 (java 1.5)"
this is what i get when i use my Php script to request a post:
127.0.0.1 - - [08/Aug/2013:15:54:23 +0200] "POST /accept.php HTTP/1.1" 200 419 "-" "-" ::1 - - [08/Aug/2013:15:54:23 +0200] "GET /testSendDat.php HTTP/1.1" 200 3330 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
and the error log from the php script shows me:
[08-Aug-2013 16:10:04 Europe/Berlin] PHP Notice: Undefined index: file_contents
So I know what that means, the array where i'm trying to read my files from are empty. But why? I have no clue. I'm guessing I'm using the wrong way to post a request but have been looking for a while now and haven't really been successful in finding what the problem could be. Help would be much obliged.
Oh and would it perhaps be easier to set up a apache tomcat server and work with httpservlets? Or would'nt that matter?
You do not create a HTTP-Request that simulate a filled in form with a file upload element in it. You create a HTTP POST Request with the file as the request body.
The body of a filled in form POST HTTP Request looks like this:
user=Fritz&age=12
It looks sumilar like the URL params of a GET HTTP Request.
A filled in form POST with a file upload in it is a multipart HTTP Request with multiple bodies. One of it has the form fields the other the upload file content.
see: how-does-http-file-upload-work
I cannot provide sample to set up your request, but this information may help you.