Http error 429 java handling? - java

I have a java program that searches a site for updates. It checks and than sleeps for a period of time set by user.
During some testing I set the sleep time to 0 secs, forgot about it and it over-requested (over-spammed?, don't know the terminology) the site. I wouldn't have noticed this if I didn't go to this site and manually checked for something.
The error I get is 429 : too many requests. I've searched the web and found out that the site should send me when is it ok to try again (retry-after header). The problem is that I either don't get this information or can't find it.
I've viewed the source via chrome and didn't find the information. Then I used getErrorStrem and didn't find the information.
In another question on stackoverflow it is said that
if everything is set up properly, you will also have received a
"Retry-after" header along with the 429 response
My knowledge of web technologies is rather limited. The program I wrote looked for patterns that I noticed viewing the source of page. My java knowledge is better, but not great.
So how do I get retry-after info? About 20 hours passed and when I tried again I got the same error.

As specified in the rfc 6585:
The 429 status code indicates that the user has sent too many
requests in a given amount of time ("rate limiting").
The response representations SHOULD include details explaining the
condition, and MAY include a Retry-After header indicating how long
to wait before making a new request.
This specific header is only optional. So if you don't see it in the headers that's because it isn't there.
Including the condition in the response representation is only a recommandation (and so not mandatory)
There is no rules regarding this limit: it depends on the server configuration.
For exact meaning of verbs commonly used in RFCs : you can read this

Related

JAVA EWS: setExtendedProperty with MapiPropertyType.Binary

I have to enhance an existing JAVA app to store deadlines into several calendar all owned (created) by the same Shared-Mailbox.
Synchronisation of the deadlines in the app and the outlook calendars is no problem.
Apart from syncing the dates the JAVA app should be able to send invitations to the specific calendar. I have used the description https://willcode4foodblog.wordpress.com/2012/04/13/understanding-sharing-invitation-requests-ews-managed-api-1-2-part-1/. I needed to port that stuff from c# to JAVA. On sendAndSaveMessage I always receive this Exception:
exchange.webservices.data.core.exception.service.remote.ServiceRequestException: The request failed. An internal server error occurred. The operation failed.
at microsoft.exchange.webservices.data.core.request.SimpleServiceRequestBase.internalExecute(SimpleServiceRequestBase.java:74)
at microsoft.exchange.webservices.data.core.request.MultiResponseServiceRequest.execute(MultiResponseServiceRequest.java:158)
at microsoft.exchange.webservices.data.core.ExchangeService.sendItem(ExchangeService.java:789)
at microsoft.exchange.webservices.data.core.service.item.EmailMessage.internalSend(EmailMessage.java:156)
at microsoft.exchange.webservices.data.core.service.item.EmailMessage.sendAndSaveCopy(EmailMessage.java:300)
Using the debugger I found out, that all Extended Properties of MapiPropertyType.Binare have a null value. That's eg.
byte[] binInitiatorEntryId = hexStringToByteArray(initiatorEntryID);
ExtendedPropertyDefinition pidLidSharingInitiatorEntryId =
new ExtendedPropertyDefinition(propertySetSharing, 0x8A09, MapiPropertyType.Binary);
invitationRequest.setExtendedProperty(pidLidSharingInitiatorEntryId, binInitiatorEntryId);
So in the debugger the extended property exists, but the value is "null".
All other fields that are noted in the example look ok in the debugger. It's just that the binaries are all "null". I also compared the sharing_metadata.xml attachment with one created by Outlook and they are identical. Ok, there are a few fields I got to play with (let's say: do I have to use a "special folder" for pidLidSharingFlavor because I didn't find any explanations in the specs on what is a special folder). But since it is obvious that a "null" value for MapiPropertyType.Binary is not correct it does not make any sense to check other possibilities.
So mainly there are two questions regarding this issue which I hope someone with a bit more experience could explain:
Question 1: Is there any special way in JAVA to store Extended Properties of MapiPropertyType.Binary?
Question 2: Is there any way to get more information on the "internal server error occured" from EWS? Even enhancing the trace does not give any more information beside the XML representation of the message.
Thanks in advance.
ChriS

Tomcat ignoring post parameters request

I have an Apache Tomcat server to read request from my webapp.
In my webapp I have a form that is submitted and posts a large number of POST parameters, around 8k~
However when I try to debug the entrypoint, where the HttpServletRequest, I always receive exactly 6841. The inputs from the form are created iterating over a number of elements, meaning that the last ones are exactly the same form as the other that are succeding
I can't show code for NDA reasons.
I discarded the frontend as an issue because with a sniffer I was able to see that the complete post param list is sent.
I believe I'm on the right track, I think Tomcat is dropping the other post params. the post size limit is well beyond the size of the request, and we don't have a post parameter count configured on server.xml (defaults to 10,000 and I don't hit that amount).
All answers I have found are about not sending parameters at all or errors being thrown, in this case they are simply ignored by Tomcat.
Increasing the number of POST parameters (not size of post) to 20,000 fixed the issue in my case. This was done in the tomcat server.xml configuration using maxParameterCount:
The maxParameterCount attribute controls the maximum number of
parameter and value pairs (GET plus POST) that can be parsed and
stored in the request. Excessive parameters are ignored. If you want
to reject such requests, configure a FailedRequestFilter.

HTTP Request (POST) field size limit & Request.BinaryRead in JSPs

First off my Java is beyond rusty and I've never done JSPs or servlets, but I'm trying to help someone else solve a problem.
A form rendered by JavaScript is posting back to a JSP.
Some of the fields in this form are over 100KB in size.
However when the form field is being retrieved on the JSP side the value of the field is being truncated to 100KB.
Now I know that there is a similar problem in ASP Request.Form which can be gotten around by using Request.BinaryRead.
Is there an equivalent in Java?
Or alternatively is there a setting in Websphere/Apache/IBM HTTP Server that gets around the same problem?
Since the posted request must be kept in-memory by the servlet container to provide the functionality required by the ServletRequest API, most servlet containers have a configurable size limit to prevent DoS attacks, since otherwise a small number of bogus clients could provoke the server to run out of memory.
It's a little bit strange if WebSphere is silently truncating the request instead of failing properly, but if this is the cause of your problem, you may find the configuration options here in the WebSphere documentation.
We have resolved the issue.
Nothing to do with web server settings as it turned out and nothing was being truncated in the post.
The form field prior to posting was being split into 102399 bytes sized chunks by JavaScript and each chunk was added to the form field as a value so it was ending up with an array of values.
Request.Form() appears to automatically concatenate these values to reproduce the single giant string but Java getParameter() does not.
Using getParameterValues() and rebuilding the string from the returned values however did the trick.
You can use getInputStream (raw bytes) or getReader (decoded character data) to read data from the request. Note how this interacts with reading the parameters. If you don't want to use a servlet, have a look at using a Filter to wrap the request.
I would expect WebSphere to reject the request rather than arbitrarily truncate data. I suspect a bug elsewhere.

Sending estimate HTTP Content-Length from Servlet etc

I often need to generate content dynamically from a servlet / restlet or whatever, and don't know the length ahead of time. If the client is a browser, the progress bar doesn't work properly, because I haven't set the Content-Length header. Is there any way of setting an estimated content length, so the progress bar works "more or less"?
No, the Content-Length value must be the exact content length:
When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. HTTP/1.1 user agents MUST notify the user when an invalid length is received and detected.
So you cannot send just an estimated Content-Length value to get a progress bar.
While I wouldn't recommend doing it you can set the content-length to an estimate of what the content is, and you will get a progress bar. As long as you can guarantee that your estimate is equal to or greater than the actual content this will generally work. If your actual content is greater than the estimate than the content will be truncated to the specified content-length.
I tested this in Firefox, IE, and Chrome without issue. The HTTP spec states that users agents MUST notify the user if the specified length doesn't match the actual length, but I have not observed this behavior with any browser I've tested.
I investigated this as an option but abandoned it due to the potential unforeseen conflicts for playing outside the bounds of the specification.
Not clear what you're asking. You can always set the Content-Length header yourself, though it needs to actually match the amount of data you're sending. The standard way to handle dynamic data where you don't know anything about the length ahead of time is to buffer the output, find the actual length, set the header, then dump the output. Not really answering what you appear to be asking, but I think what you're asking is impossible.
The only way to do this (as decribed in an RFC) is not to set the Content-Length header, ie. the response header does not contain a Content-Length line. In this case the browser does not know how long the body is, so the server "tells" the browser that all of the body has been sent by closing the connection.
I'm not sure whether the Java container will automatically close the connection in this case or if you can do it yourself via some kind of Filter.
To answer your question: I do not think that it is possible to give the browser an estimate.
If the length of your content is not known beforehand you can use the “chunked” content encoding (as per HTTP version 1.1). This will not solve your progress bar problem, though—and there’s just no way to make it work unless you know how much content you are going to send.

Multiple questions in Java (Validating URLs, adding applications to startup, etc.)

I have an application to build in Java, and I've got some questions to put.
Is there some way to know if one URL of a webpage is real? The user enters the URL and I have to test if it's real, or not.
How can I konw if one webpage has changes since one date, or what is the date of the last update?
In java how can I put an application running on pc boot, the application must run since the user turns on the computer.
I'm not sure what kind of application you want to build. I'll assume it's a desktop application. In order to check if a URL exists, you should make a HTTP HEAD Request, and parse the results. HEAD can be used to check if the page has been modified. In order for an application to start when the PC boots, you have to add a registry entry under Windows; this process is explained here
To check whether a url is valid you could try using a regular expression (regex for urls).
To know if a webpage has changed you can take a look at the http headers (reading http headers in java).
You can't make the program startup automatically on boot, the user must do that. However, you can write code to help the user set the program as startup app; this however depends on the operating system.
I'm not sure what you mean by "real". If you mean "valid", then you can just construct a java.net.URL from a String and catch the resulting MalformedURLException if it's not valid. If you mean that there's actually something there, you could issue an HTTP HEAD request like Geo says, or you could just retrieve the content. HTTPUnit is particularly handy for retrieving web content.
HTTP headers may indicate when the content has changed, as nan suggested above. If you don't want to count on that you can just retrieve the page and store it, or even better, store a hash of the page content. See DigestOutputStream for generating a hash. On a subsequent check for changes you would simply compare the new hash with the the one you stored last time.
Nan is right about start on boot. What OS are you targeting?

Categories