We Keep Coding

endpoint_failure (context canceled) resulting in duplicate call to REST Endpoint

Am invoking an REST endpoint from another service using restTemplate.exchange.
The endpoint that receives the request invokes DB and fetches around 1.5 mil records and stores them in another DB.
Now am getting below x_cf_routererror:"endpoint_failure (context canceled)" after invoking the DB. I get this error in about 120+ seconds and process continues as is.
After this error I see another call being made the same endpoint and this is resulting duplicates in target DB.
Not sure why this is happening, I do not have any retry mechanism in place and the restTempalte timeout is set to 300 at client service that invokes.
Has someone faced this issue? whats causing this endpoint_failure (context canceled) and duplicate invocation of endpoint.
Appreciate your help in this.
Log snippet:
2022-05-12T08:57:18.840-04:00 [APP/PROC/WEB/0] [OUT] 2022-05-12 12:57:18.840 INFO 28 --- [nio-8080-exec-4]
Controller1 : Request received to load all timecard information::RequestedTime=12:57:18.840
2022-05-12T08:59:21.530-04:00 [RTR/17] [OUT] - [2022-05-12T12:57:18.829182975Z] "GET HTTP/1.1" 499 0 22 "-" "Java/1.8.0_332" "" "1" x_forwarded_for:"" x_forwarded_proto:"https" vcap_request_id:"" response_time:122.701301 gorouter_time:0.000164 app_id:"" app_index:"0" instance_id:"" x_cf_routererror:"endpoint_failure (context canceled)" x_b3_traceid:"" x_b3_spanid:"" x_b3_parentspanid:"-" b3:"599552bb012c2adc60adef7187a865e7-60adef7187a865e7"
**Below is the duplicate call**
2022-05-12T08:59:21.777-04:00 [APP/PROC/WEB/0] [OUT] 2022-05-12 12:59:21.777 INFO 28 --- [nio-8080-exec-2]
Controller1 : Request received to load all timecard information::RequestedTime=12:59:21.777
Thanks,
S

The error 499 (or 502) is returned by PCF gorouter, instead of your web app [APP/PROC/WEB/0].
PCF gorouter will retry with same HTTP request for particular error cases.
For more details please refer to: https://docs.cloudfoundry.org/adminguide/troubleshooting-router-error-responses.html

Error 499 usually mean that call is taking too much of time and client closed the connection. It has been noticed many times when your app is not able to get a response within specific time (usually 2min time limit set). You might need to see if there is any DB issue or something else that delaying the expected response.

Java Scribe library Twitter OAuth1 request token 401 unauthorized

I have been trying to use different Java OAuth libraries to get request and access token from Twitter API. Finally, I have come to use Scribe as one of the most stable libraries for this matter. However, I have not been able to get this to work with Twitter on the first step which is getting the access token:
twitterOAuthService = new ServiceBuilder(twitterOauthConsumerKey)
.apiSecret(twitterOauthConsumerKey).debug().callback(callback).build(
TwitterApi.instance());
OAuth1RequestToken requestToken = twitterOAuthService.getRequestToken();
It throws the following exception:
Response{code=401, message='Authorization Required', body='null', headers={date=Tue, 05 Nov 2019 07:06:34 GMT, null=HTTP/1.1 401 Authorization Required, server=tsa_l, content-length=64, expires=Tue, 31 Mar 1981 05:00:00 GMT, x-response-time=166, x-frame-options=SAMEORIGIN, www-authenticate=OAuth realm="https://api.twitter.com", x-transaction=00e2c19300b2d1ab, strict-transport-security=max-age=631138519, pragma=no-cache, set-cookie=guest_id=v1%3A157293759473549885; Max-Age=63072000; Expires=Thu, 4 Nov 2021 07:06:34 GMT; Path=/; Domain=.twitter.com, last-modified=Tue, 05 Nov 2019 07:06:34 GMT, x-xss-protection=0, x-content-type-options=nosniff, content-disposition=attachment; filename=json.json, x-connection-hash=badf004485a6d713a29ed3f90de9e981, x-twitter-response-tags=BouncerCompliant, content-type=application/json; charset=utf-8, cache-control=no-cache, no-store, must-revalidate, pre-check=0, post-check=0, status=401 Unauthorized}}
I can see that the Authorization header is set properly:
Authorization -> OAuth oauth_callback="http%253A%252F%252F127.0.0.1%253A3000%252Fsettings", oauth_consumer_key="8hupaLbXA2s792EzU8DIE6K", oauth_nonce="2929183554", oauth_signature="s%2BRMAninTESJqJg92BuEA8h3E%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1572936780", oauth_version="1.0"
I have tested the below approaches as they were suggested in different places:
Ensure Authorization header is sorted alphabetically!
Ensure that the timestamp is set properly
Ensure that the consumer secret and consumer key are set properly
Ensure that the encoding of oauth_signature (percent encoding) and call_back is set properly
I have run out of any other scenarios and I am not even sure if I can use Scribe library for Twitter. I would appreciate it if someone could help me to address this issue.

Retrofit client and response with Transfer-Encoding: chunked

I'm working on a Android sample app that get film list from http://www.omdbapi.com/.
The REST service is:
http://www.omdbapi.com/?s=star&apikey=d497e644
I'm using Retrofit to write the client.
public interface OmdbApi {
#Streaming
#GET("./")
#Headers({"Cache-control: no-cache"})
Call<Search> search(#Query("s") String search, #Query("apikey") String apiKey);
#Streaming
#GET("./")
#Headers({"Cache-control: no-cache"})
Call<FilmDetail> getFilm(#Query("i") String uid, #Query("apikey") String apiKey);
}
The complete source code is available here.
When I run the application, I obtain the following response (taken from logcat):
OK http://www.omdbapi.com/?s=star&apikey=d497e644 (108ms)
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: public, max-age=86400
Expires: Sat, 26 May 2018 14:28:18 GMT
Last-Modified: Fri, 25 May 2018 05:39:04 GMT
Vary: *, Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: *
CF-Cache-Status: HIT
Server: cloudflare
CF-RAY: 4208b00c817b3db9-MXP
Connection: Keep-Alive
And the following error:
java.net.ProtocolException: unexpected end of stream
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:455)
at okio.RealBufferedSource.read(RealBufferedSource.java:47)
okio.RealBufferedSource.exhausted(RealBufferedSource.java:57)
okio.InflaterSource.refill(InflaterSource.java:102)
okio.InflaterSource.read(InflaterSource.java:62)
okio.GzipSource.read(GzipSource.java:80)
okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:237)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180)
retrofit2.ExecutorCallAdapterFactory$ExecutorCallbackCall.execute(ExecutorCallAdapterFactory.java:91)
com.abubusoft.filmfinder.service.repository.FilmRepository.lambda$findFilm$0$FilmRepository(FilmRepository.java:18)
com.abubusoft.filmfinder.service.repository.FilmRepository$$Lambda$0.run(Unknown Source:20)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
After a bit of investigation, I found that the problem is that the response has Transfer-Encoding: chunked. How can I resolve this problem?
Thank you.

After more investigation:
I remove the #Streaming annotations.. they aren't useful.
Tests that I did was on an emulator on a PC behind a firewall.
OkHttp had some problem about it many years ago, but now chunked transfer is completely managed.
On the same machine, invoking the same URL in a browser does not present any problems
I finally try on another machine, with a JUnit test and in Android emulator and I had no problem.
At last, I'm sure that code works, so is the environment that I used to develop the app has some problem with transfer-encoding chunked.
Complete source code is available here:
android app
junit test
I have to find exactly what is the problem, I'm sure now that it not Retrofit or my client definition or OkHttp.
I hope my experience can help other developers.
Byez

Google Drive API responds with 416 HTTP code

For some reason the previously working code stopped working and server started to respond with 416.
Here are the logs of HTTP client during failing interaction:
-------------- REQUEST --------------
GET https://www.googleapis.com/drive/v3/files/0B02Nopv3SQOvOVNKaDIwTEZ3MHd?alt=media
Accept-Encoding: gzip
Authorization: <Not Logged>
Range: bytes=0-33554431
User-Agent: My app Google-API-Java-Client Google-HTTP-Java-Client/1.22.0 (gzip)
-------------- RESPONSE --------------
HTTP/1.1 416 Requested range not satisfiable
Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35"
Server: UploadServer
Cache-Control: private, max-age=0
Content-Range: bytes */0
X-GUploader-UploadID: AEnB2UqBx9B09Lnr8tG761gdoz3DkhHSNO_OzHh1LkU6B2908v17rnBGQZSNW4ZVTjbRdFtvPWWIqZGdtSrTo6ZWN7YW9nxf6d
Vary: X-Origin
Vary: Origin
Expires: Mon, 11 Sep 2017 15:23:20 GMT
Content-Length: 225
Date: Mon, 11 Sep 2017 15:23:20 GMT
Content-Type: application/json; charset=UTF-8
I was trying to download a file which is around 200000 bytes, so I thought meaning of "chuck size" changed somewhere, so it could not give 33554431 bytes of a 282177 byte file. Tried changing that to a smaller value, but no success.
Drive.Files.Get get = drive.files().get(file.getId())
MediaHttpDownloader downloader = get.getMediaHttpDownloader()
downloader.directDownloadEnabled = false
localFile.newOutputStream()
get.executeMediaAndDownloadTo(stream)
Direct download does not work either, it just downloads "0" bytes.
Does anyone know how to overcome this issue?

416 Range Not
Satisfiable
error means the server is not able to serve the requested ranges. The
most likely reason is that the document doesn't contain such ranges,
or that the Range header value, though syntactically correct, doesn't
make sense.
One of the resolutions that may provide from this forum is to:
Add "Accept-Ranges: none" to our response headers.

It appeared to be a web interface when using Firefox. It uploaded "empty" files in certain cases.
https://productforums.google.com/forum/#!topic/drive/S03wEknc75g;context-place=forum/drive

Groovy HTTP Builder: Empty POST response causes SAXParseException

I am implementing REST client for YouTrack API. I am getting an exception whenever my POST requests return empty response (there's just a valid header). The HTTP status code is 200.
That's the stacktrace:
[Fatal Error] :-1:-1: Premature end of file.
gru 03, 2013 10:34:49 AM groovyx.net.http.HTTPBuilder doRequest
WARNING: Error parsing 'application/xml; charset=UTF-8' response
org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at groovy.util.XmlSlurper.parse(XmlSlurper.java:181)
at groovy.util.XmlSlurper.parse(XmlSlurper.java:234)
at groovyx.net.http.ParserRegistry.parseXML(ParserRegistry.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1085)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:952)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
at groovy.lang.Closure.call(Closure.java:423)
at groovy.lang.Closure.call(Closure.java:439)
at groovyx.net.http.HTTPBuilder.parseResponse(HTTPBuilder.java:561)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:494)
at groovyx.net.http.RESTClient.post(RESTClient.java:140)
at groovyx.net.http.RESTClient$post.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at groovy.YouTrackRest.releaseVersion(YouTrackRest.groovy:66)
at groovy.YouTrackRest$releaseVersion$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at YouTrack.run(YouTrack.groovy:13)
[...]
I have debugged and found out what's the direct cause for that. I noticed that in groovyx.net.http.HTTPBuilder:492 before parsing it is checked whether response is empty but in my case the call entity.getContentLength() returns -1. After Apache documentation for HttpEntity
the number of bytes of the content, or a negative number if unknown
My client implementation is very plain. I have worked the problem around by catching the HttpResponseException and asserting statusCode 200. I am still getting the stacktrace in my console, but that's probably just the logger setting.
My implementation:
import groovyx.net.http.RESTClient
class YouTrackRest extends RESTClient {
[...]
def activateVersion(versionNumber) {
try {
post(
path: getVersionPath(versionNumber),
body: [
colorIndex: ACTIVE_COLOR_INDEX,
released: false,
releaseDate: null,
archived: false,
newName: versionNumber
]
)
} catch (HttpResponseException ex) {
assert ex.response.status == 200
}
}
EDIT:
These are the response headers
Server: nginx
Date: Fri, 06 Dec 2013 10:36:10 GMT
Content-Type: application/xml; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: JSESSIONID=ny2s3cxwsvveya6vgwj2fmig;Path=/youtrack
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache, no-store, no-transform, must-revalidate
Present Transfer-Encoding: chunked header explains missing 'Content-Length' header. That has to be the reason for unknown Content Length.
Now, according to Wikipedia The data transfer is terminated by a final chunk of length zero. So maybe is this case the first chunk is also the last one?
Both, the Web Server and the Http client library are third party for me. I would like to raise an issue by whoever misbehaves:
either YouTrack REST WS promisses XML and returns nothing
or groovyx.net.http.HTTPBuilder fails to process chunked encoded responses.
I will be very helpful for some expertise to help me understand the problem.

I just ran into this same problem using httpbuilder. I was able to work around it with:
def service = new HTTPBuilder(...)
service.request(Method.POST, ContentType.JSON) {
uri.path = ...
body = ...
response.'200' = { resp ->
logInfo resp.statusLine
}
response.success = { resp, result ->
logInfo resp.statusLine
}
response.failure = { resp ->
logError resp.statusLine
}
}
Essentially do not try to parse the body, if response is 200.

Apparently the problem was caused by non-standard WS response: HTTP status 200 and empty response body. Either HTTP status 204 is expected, or valid XML body and status 200.
I have reported an issue at WS providers. Hopefully it will get resolved. I am marking the question as answered
Thanks #tim_yates and #Ian Roberts for your helpful comments.