I've created a program that parses html pages. I use jsoup connect function within a callable class inside ThreadPool. The problem is that I'm connecting to the same website and with a thread pool size of 5+, I get IO Exceptions - 400 errors.
How do I not make that happen?
If you're getting a 400 HTTP response, check the content of the response for an error message. A 400 means a bad request of some kind: you didn't include all the required information or included malformed information. Some people also use it as kind of a catch-all for when the client did something they don't like. If you're making lots of different requests, examine the ones that caused 400's to see if there's something wrong with them. If they all look right or if you're sending the same requests repeatedly, then maybe the site you're hitting has some kind of rate limiting that disallows you from making too many concurrent requests or too many requests within a particular timeframe. If it's something like that, I'd expect there to be a message in the response telling you what's going on.
Related
I have been tasked with determining why certain tests are failing in Postman's collection runner. I found this question helpful but it does not address the scope of my issue.
This post suggests I could be sending the request with invalid JSON, but I did not write these tests, so I am unsure of how to check that.
My collection runs 423 GET/POST tests:
280 pass
143 fail
100% of all GET tests pass, as well as some POST tests.
However, all tests that fail are POST tests. Of these:
Almost all of them get a 500 Internal Server Error
They do not log a Response Header or Response Body
So far, I have ensured the app server is listening on the correct ports, and I've restarted the server a few times. Beyond this I'm unsure of how to proceed; most of my experience is in Java desktop applications but I'm now tackling this web application.
What steps should I try next?
A 500 error could be almost anything, and in general you would need to look at server/app logs to figure out what happened. The error could have been thrown because of something unexpected/catastrophic or simply because the server (or rather the programmer/admin) didn't want to expose any more information.
With that said, I would also make sure that your POST request is being sent to an endpoint that is listening/expecting a POST and that the request is properly formatted.
I am consuming a REST webservice, which is hosted on a remote server.
Now, the web service's POST, PUT and DELETE methods are blocked by the server's firewall. So I have only GET method left.
Is there a way I can use GET to work as POST or PUT or DELETE ?
I am using jersey api for consuming the services.
https://groups.yahoo.com/neo/groups/rest-discuss/conversations/messages/9962
Yes. In other words, any HTTP request message is allowed to contain
a message body, and thus must parse messages with that in mind.
Server semantics for GET, however, are restricted such that a body,
if any, has no semantic meaning to the request. The requirements
on parsing are separate from the requirements on method semantics.
So, yes, you can send a body with GET, and no, it is never useful
to do so.
This is part of the layered design of HTTP/1.1 that will become
clear again once the spec is partitioned (work in progress).
....Roy
This means you can send a message body with your GET request but it would have no meaning. If your server firewall doesn't allow you to receive requests other than GET you should try to talk to your admin, change the server and if that's not possible due to company reasons you should escalate it via your manager.
edit: as Kayaman said it's not your job to do that. if you are not given the right infrastructure, you just can't work. if you are not given a computer nobody would expect you to be able to code.
For an assignment, I have written a server that services HTML files and I am supposed to use my web browser as a test client. I am also told that if there is a request for a file that doesn't exist i should send the following
HTTP/1.1 404 Not Found\r\n\r\n and if anything else goes wrong
HTTP/1.1 500 Internal Server Error\r\n\r\n"
I have run tests that should cause those to be sent, but nothing occurs in my browser window? Should I be getting any visual feedback from sending such a request?
In an HTML response there should only be one CRLF (carriage return and line feed) after each line. So you can first remove the extra \r\n.
Also on the second line you can send an HTML response back to the client saying what the error was if you want to show an error. This is normally what a typical web server does where it has its default error page if one is not defined. If you are not sending any HTTP headers, then you can insert the HTML body you want to send back such as <h1> No page found</h1> as the second line.
To include an "entity" in an http response
HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 10
01234576789
each line is ended by CRLF. Be sure to count the Content-Length correctly.
I see that this is a learning exercise, and understand that you may have been told to implement the server this way. But bearing that in mind ...
This is the wrong way to implement a web server / service. The right way is to find an existing implementation and build your service on top of that.
You could use a Java EE web container; i.e. something that implements the Servlets spec.
You could use a non-servlet framework (like Grizzly).
You could build on top of a server-side HTTP protocol stack; e.g. using Apache the HttpComponent library.
Building a web server from the ground up is a lot of work if you are going to do it properly. And the chances that you won't do it properly; i.e. you won't implement your service according how the HTTP spec says a server should behave. You will leave things out, do things the wrong way, etc.
Please don't do it. There are already too many broken (i.e. non-compliant) web servers out there. We don't need more.
And if you DO decide to implement HTTP from the ground up, then you (YOU) need to thoroughly read and understand the HTTP spec. And you (YOU) need to do your own basic research on how browsers implement the client side of the spec ... and what you therefore need to do on the server side to make browsers behave "normally".
How can I make a java-based application server reply with an empty-valued response header, like this?
content-length:\r\n
Unfortunately when I call
response.setHeader("Content-Length", len)
where len is either an empty string or null, the response will not include the header.
I've checked the HttpServletResponse and HttpServletResponseWrapper javadocs but couldn't figure out what could be overriden to provide my custom behaviour.
Background
I'm building a testing application that is supposed to emulate badly-behaved HTTP server scenarios. The application is supposed to reply to requests with preset pages and HTTP headers, including malformed ones like the above case.
The application is written in grails.
I'm building a testing application that is supposed to emulate badly-behaved HTTP server scenarios.
In such a case, attempting to get a well-behaving server to mimic such behavior is a bad idea. If you need to mimic a bad server, or a particular set of scenarios you wish to test, then you may do one of the following:
write a custom application that listens on a particular port (using the ServerSocket class) that will respond with malformed HTTP headers. Using HTTP libraries may not help, for libraries may have code to detect erroneous conditions and correct them automatically.
use a HTTP proxy that is capable of intercepting responses and allows for modifications of these responses. You will find several if you Google for "http debugging proxy", but if you haven't heard of any, I would suggest looking at Fiddler, WebScarab or Burp.
You can try a tool like SoapUI or Fiddler with it's Firefox extension. I havent tried setting a malformed header with them but I wouldn't be suprised if you could.
Something not clear for me: your application is written in Grails, but you are discussing of javadocs... Well, I suppose you try to create a bad server in JAVA...
As you said, answering with "Content-Length:\r\n" is not legal for HTTP. You must put an integer value or discard the header. I think setHeader() helps you to avoid to produce an illegal HTTP message.
You can workaround this way creating manually the headers (you can write directly to the socket without using the setHeader blocks).
Other solution is to create a filter (in addition of your servlet) with your own implementation of HttpServletResponse. You will pass this implementation to the servlet.
Is it possible to send "100 Continue" HTTP status code, and then later some other status code after processing entire request using Java Servlet API (HttpServletResponse)?
I can't find any definitive "No" answer, although API doesn't seem to support it.
I assume you mean "100 Continue".
The answer is: no, you can't (at least not the way it's intended, as provisional response). In general, the servlet engine will do it automatically, when the request requires it. Of course this makes it impossibe for the servlet to prevent sending the 100 status -- this issue is a known problem in the Servlet API, and has been known for what feels like eons now.
I know that Jetty will wait until getReader() or getInputStream() is called before it sends a 100. I think this is the behavior you are looking for. I don't know what Tomcat does.
Did you mean to ask How do I send a status code before the complete request is received, to interrupt an in-progress request due to a missing header field? It seems that's not possible with standard servlets.
What server are you using?
Some server's servlet extensions may allow this, e.g. Tomcat's Comet servlet might send EventType.BEGIN once the headers are available to process, which may allow you to interrupt a PUT that doesn't have the correct authentication.
Alternatively your server might have a plugin to reject requests based on headers.
Do you mean status code 100 ?
The API does support sending SC_CONTINUE.