I have been trying to put together some code that will- among other things - upload files to a Sharepoint site that uses NTLM authentication. Earlier versions of the code were single threaded, and worked perfectly. They uploaded the file exactly as expected without the slightest issue. However, I eventually tried to multi-thread this application, so that it could upload many files at once, while still going about the rest of its business.
However when I tried to multithread the code, it fails every single time, throwing an IndexOutOfBoundsException. This is singularly unhelpful to me in diagnosing the actual cause of the problem.
In case you are wondering, if I change out the CachedThreadExecutor for a SingleThreadExecutor - forcing the code bask to a single-threaded state - it once again works fine.
Creating the executor and connection manager, and constructing threads:
class OrderProcessor implements Runnable {
//Other variables for object
private final ExecutorService executorService = Executors
.newCachedThreadPool();
// .newSingleThreadExecutor();
private HttpClientConnectionManager conManager;
private void setup() {
//always called before execution of anything else in object
conManager = new PoolingHttpClientConnectionManager();
}
//lots of other code
}
The actual code for submitting the threads is complicated, so this version is somewhat simplified, but gets the point across.
for(Request request : requests){
//Do other stuff
simpleSubmitFile(request);
//Do other stuff
}
Here is the simplified file submission method
public Future<Boolean> simpleSubmitFile(Request request){
transferer = new SharePointTransferer(extractionRequest, conManager);
Future<Boolean> future = executorService.submit(transferer);
return future;
}
SharePointTransferer code
//actual values scrubbed
private final String USERNAME = "";
private final String PASSWORD = "";
private final String DOMAIN = "";
private final File sourceFile;
private final String destinationAddress;
private final CloseableHttpClient client;
public SharePointTransferer(final Request extractionRequest, HttpClientConnectionManager conManager) {
super(extractionRequest);
this.sourceFile = this.extractionRequest.getFile();
this.destinationAddress = this.extractionRequest.getDestinationAddress();
this.client = HttpClients.custom()
.setConnectionManager(conManager).build();
}
public Boolean call() throws Exception {
String httpAddress = correctSharePointAddress(destinationAddress);
HttpPut put = new HttpPut(httpAddress + sourceFile.getName());
// construct basic request
put.setEntity(new FileEntity(sourceFile));
HttpClientContext context = HttpClientContext.create();
// set credentials for the SharePoint login
CredentialsProvider credProvider = new BasicCredentialsProvider();
credProvider.setCredentials(AuthScope.ANY, new NTCredentials(USERNAME,
PASSWORD, "", DOMAIN));
context.setCredentialsProvider(credProvider);
// execute request
try {
HttpResponse response = client.execute(put, context);
logger.info("response code was: "
+ response.getStatusLine().getStatusCode());
if (response.getStatusLine().getStatusCode() != 201) {
throw new FileTransferException(
"Could not upload file. Http response code 201 expected."
+ "\nActual status code: "
+ response.getStatusLine().getStatusCode());
}
} catch (ClientProtocolException e) {
throw new FileTransferException(
"Exception Occurred while Transferring file "
+ sourceFile.getName(), e);
} catch (IOException e) {
throw new FileTransferException(
"Exception Occurred while Transferring file "
+ sourceFile.getName(), e);
}finally{
logger.info("deleting source file: " + sourceFile.getName());
sourceFile.delete();
client.close();
}
logger.info("successfully transfered file: "+sourceFile.getName());
return true;
}
If I submit multiple files it throws essentially the exact same exception for all of the files. The trace is below
Exception Stack Trace
2015-04-16 11:49:26 ERROR OrderProcessor:224 - error processing file: FILE_NAME_SCRUBBED
PACKAGE_SCRUBBED.FileProcessingException: Could not process file: FILE_NAME_SCRUBBED
at PACKAGE_SCRUBBED.OrderProcessor.finishProcessingOrder(OrderProcessor.java:223)
at PACKAGE_SCRUBBED.OrderProcessor.run(OrderProcessor.java:124)
at PACKAGE_SCRUBBED.FileTransferDaemon.process(FileTransferDaemon.java:48)
at PACKAGE_SCRUBBED.FileTransferDaemon.start(FileTransferDaemon.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 41
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at PACKAGE_SCRUBBED.OrderProcessor.finishProcessingOrder(OrderProcessor.java:208)
... 8 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 41
at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte(NTLMEngineImpl.java:924)
at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addUShort(NTLMEngineImpl.java:946)
at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getResponse(NTLMEngineImpl.java:1052)
at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTLMEngineImpl.java:148)
at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NTLMEngineImpl.java:1641)
at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme.java:139)
at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSchemeBase.java:138)
at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthenticator.java:239)
at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResponse(HttpAuthenticator.java:202)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:262)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at PACKAGE_SCRUBBED.SharePointTransferer.call(SharePointTransferer.java:74)
at PACKAGE_SCRUBBED.SharePointTransferer.call(SharePointTransferer.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
If anyone can figure out what is causing this problem, I would greatly appreciate it.
EDIT: I managed to find a workaround that fixes the issue for me, but would still appreciate an explanation of exactly what is going on.
this is a bug, solved in httpclient version 4.5.2
http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt
Release 4.5.2
Changelog:
[HTTPCLIENT-1715] NTLMEngineImpl#Type1Message not thread safe but declared as a constant. Contributed by Olivier Lafontaine , Gary Gregory
You can't reuse nor HttpClientContext neither NTLMScheme in a concurrent environment because they are both marked as #NotThreadSafe (see javadoc).
In my environment I got the same error, solved with something like:
synchronized(context) {
HttpResponse response = client.execute(put, context);
}
The authenticated context is reused, but one thread at time.
I eventually managed to solve this problem by setting the number of connections per route to 1, as below.
conManager.setDefaultMaxPerRoute(1);
I'm still not exactly sure why the problem occured, or what the proper way to fix this is, but this solution worked for me.
Related
I am not able to print the response from a Soap Webservice.
Seen few solutions by editing the generated stub code. But I cant edit the generated code as it gets restored to original form on every build. Looking for a solution where I can get the solution printed without change in generated code.
I am consuming the SOAP service from a Spring Boot microservice.
ServiceContext serviceConxt = omsSchedulingService._getServiceClient().getServiceContext();
OperationContext operationContext = serviceConxt.getLastOperationContext();
MessageContext inMessageContext = operationContext.getMessageContext("Out");
log.info(inMessageContext.getEnvelope().toString());
You can add a message handler for the soap message.
Then once you intercept the message with the handler, you can print out the response.
You will need to add the handler to the handler chain, depending on your project you can do that programatically or with config.
final class MyMessageHandler implements SOAPHandler<SOAPMessageContext>{
#Override
public void close(MessageContext context) {
handle(context);
}
private boolean handle(MessageContext context) {
if (context != null) {
try {
Object httpResponseCodeObj = context.get(SOAPMessageContext.HTTP_RESPONSE_CODE);
if (httpResponseCodeObj instanceof Integer)
httpResponseCode = ((Integer) httpResponseCodeObj).intValue();
if (context instanceof SOAPMessageContext) {
SOAPMessage message = ((SOAPMessageContext) context).getMessage();
ByteArrayOutputStream byteOut = new ByteArrayOutputStream(512);
message.writeTo(byteOut);
String messageStr = byteOut.toString(getCharacterEncoding(message));
boolean outbound = Boolean.TRUE.equals(context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY));
Logger.info(loggingPrefix, outbound ? "SOAP request: " : "SOAP response: ", replaceNewLines(messageStr));
}
} catch (SOAPException e) {
Logger.error(e, loggingPrefix, "SOAPException: ", e.getMessage(), NEWLINE);
} catch (IOException e) {
Logger.error(e, loggingPrefix, "IOException: ", e.getMessage(), NEWLINE);
}
}
return true;
}
}
If you don´t want to implement an interceptor the easiest way is to use the logging via vm arguments:
JAVA_OPTS=-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.showdatetime=true -Dorg.apache.commons.logging.simplelog.log.httpclient.wire=debug -Dorg.apache.commons.logging.simplelog.log.org.apache.commons.httpclient=debug
This way you should see the logging of your request / response with headers in console.
First you can get AxisConfiguration from client stub.
AxisConfiguration axisConf = stub._getServiceClient().getAxisConfiguration();
Processing incoming and outgoing messages is divided into phases. There is a list of phases (a flow) which is processed when everything works correctly (without errors) and also another for situations when some fault occurs e.g. when an exception is thrown during message processing. Every flow maybe incoming or outgoing so there are 4 flows altogether.
List<Phase> phasesIn = axisConf.getInFlowPhases(); // normal incoming communication i.e. response from webservice
List<Phase> phasesOut = axisConf.getOutFlowPhases(); // normal outgoing communication
List<Phase> phasesFaultIn = axisConf.getInFaultFlowPhases(); // faulty incoming communication e.g. when an exception occurs during message processing
List<Phase> phasesFaultOut = axisConf.getOutFaultFlowPhases(); // faulty outgoing communication
Some but not all phase names are defined in org.apache.axis2.phaseresolver.PhaseMetadata.
For example "Security" phase processed in Rampart module (module for Web Service Security) won't be found in PhaseMetadata.
You can add a handler to every phase, e.g.
for (Phase p : phasesOut) {
if (PhaseMetadata.PHASE_TRANSPORT_OUT.equals(p.getName())) {
p.addHandler(new MessageContentLoggerHandler());
}
}
Handler is a class which extends org.apache.axis2.handlers.AbstractHandler.
You just have to implement
public InvocationResponse invoke(MessageContext msgContext).
There you have access to MessageContext. Of course, you can get whole SOAP envelope like this:
msgContext.getEnvelope().toString()
and for example print it to your logs or save as a separate file.
Remember to put
return InvocationResponse.CONTINUE;
at the end of invoke method for a situation when handler processes the message successfully. Otherwise processing stops in this handler and a whole process won't get to any another phase.
If you need to see whole message with WSS headers, you can add your own phase. For example this adds your custom phase as the last in processing of outgoing message (so also after Rampart's security phase)
Phase phase = new Phase("SomePhase");
phase.addHandler(new SomeCustomHandler());
axisConf.getOutFlowPhases().add(phase);
Of course logging (and exposing in any other way) security headers in production environment is a very bad idea. Do it only for debugging purposes in some test environment.
I am relatively new to App Engine. I do not understand how to asynchronously make an HTTP request with Java. I would have thought that this was a very simple matter using Thread and Runnable. But it seems that App Engine does not permit their use.
public Hashtable someApiMethod(..) {
SomeEntity entity = new SomeEntity(..);
ObjectifyService.ofy().save().entity(entity).now();
makeSomeHttpRequest(entity);
return launchResponse;
}
My question is this: how do I implement the method makeSomeHttpRequest(..) such that it returns without waiting for the URLFetchService.fetchAsync to return. I have tried the following without success:
protected void makeSomeHttpRequest(SomeEntity entity) {
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
try {
URL url = new URL("https://www.example.com");
Future future = fetcher.fetchAsync(url);
HTTPResponse response = (HTTPResponse) future.get();
byte[] content = response.getContent();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write(content);
String responseString = new String(bos.toByteArray());
int responseCode = response.getResponseCode();
// Here I will do something with the responseCode and responseString
if (responseCode == 200) entity.someValue = responseString;
} catch (IOException e) {
// handle this
} catch (InterruptedException e) {
// handle this
} catch (ExecutionException e) {
// handle this
}
}
Really what I am trying to do is perform this HTTP request without forcing the method someApiMethod to wait for the response.
Few things:
First. Future doesn't work this way. Method .get waits for result of Feature execution, so basically you're stopping current thread until other thread finishes its execution. You're making it synchronous, it doesn't make any sense. Usually you call .get much later, when all other work in current thread is finished
Second. Threads in Appengine are limited to current request, you have to complete all async processing during current request. So updating an entity in such way doesn't make much sense, it's still bounded to current request. I mean in your case makeSomeHttpRequest(entity); should work faster than return launchResponse;
What you really need is to send this data to TaskQueue and make processing of SomeEntity entity from there (but don't send entity itself, just send ID and load by id from queue task). Basically it's going to be a new request handler (servlet/controller/etc) that should load entity by id, execute makeSomeHttpRequest (synchronously) and return http status 200.
See TaskQueue docs: https://cloud.google.com/appengine/docs/java/taskqueue/
You need Push Queue most likely: https://cloud.google.com/appengine/docs/java/taskqueue/overview-push
In my servlet, I am executing the following code:
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(3000).setConnectTimeout(3000).build();
CloseableHttpAsyncClient client = HttpAsyncClients.custom().setDefaultRequestConfig(requestConfig).build();
try
{
client.start();
for (String request : preparedURLs)
{
client.execute(new HttpGet(request), new FutureCallback<HttpResponse>()
{
public void failed(Exception ex)
{
System.out.println("\n\nRequest Failed Due to : " + ex.getMessage());
}
public void completed(HttpResponse response)
{
System.out.println("\n\nRequest COMPLETED");
}
public void cancelled()
{
System.out.println("\n\nRequest CANCELLED");
}
});
System.out.println("\n\n" + request);
}
}
finally
{
System.out.println("\n*** Finally called ***\n\n");
client.close();
}
But I don't get any response at all. Following is printed in my catalina.out:
http://localhost:8080/servlet/?ps=true
http://localhost:8080/servlet/?ps=true
http://localhost:8080/servlet/?ps=false
*** Finally called ***
I have wrote this code, taking this as example from apache's official site.
I have only omitted the latch part. Is this some thing to do with latch?
If possible please explain the reason of failure too.
This latch that you have removed is actually the synchronization part of the example.
The idea of async client is to make some requests and wait for the responses on other thread. By removing the synchronization the execution directly passes to the finally block instead of waiting the responses and closes the HttpAsyncClient.
In order to get it back working add the latch code from the example.
This is the correct behaviour. You are creating a socket with a conditional Future, and then you proceed to close it without waiting for it.
The example code you linked use a "latch" variable to do so.
You may move the close code inside future callback, but probably this will confuse your waring.
Or you can use the same system of the example, with a shared synchronised counter (maybe just a final AtomicBoolean triggered by an aswer) and wait for it before continuing with your program flow, but that destroy the idea of using async call
When trying to post a RESTful Service via docker-client to my private docker registry i get that error. The confusing thing about that is that the input stream changes itself as you can see here. Its made somewhere in the background of jersey but i cannot find the cause for it. I guess the problem occurs somewhere inside the jersey logic
Method
public static Service initService(String imageId) {
final com.spotify.docker.client.DockerClient docker = new DefaultDockerClient(
"http://10.###.###.143:2375");
String s = null;
try {
s = ("10.###.###.143:5000/user/ipatest&tag=latest");
docker.pull(s);
System.out.println(docker.toString());
} catch (DockerException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Exception
com.spotify.docker.client.DockerRequestException: Request error: POST http://10.###.###.143:2375/v1.12/images/create?fromImage=10.###.###.143%3A5000%2Fuser%2Fipatest%26tag%3Dlatest: 500
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:563)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:544)
at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:345)
at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:329)
at de.fhg.ipa.vfk.eapps.commoniaas.docker.DockerServiceMgmt.initService(DockerServiceMgmt.java:43)
at de.fhg.ipa.vfk.eapps.commoniaas.docker.DockerServiceMgmt.main(DockerServiceMgmt.java:163)
Caused by: com.sun.jersey.api.client.UniformInterfaceException: POST http://10.###.###.143:2375/v1.12/images/create?fromImage=10.###.###.143%3A5000%2Fuser%2Fipatest%26tag%3Dlatest returned a response status of 500 Internal Server Error
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:688)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:540)
... 4 more
It must have something to do with queryParams(String params) which belongs to WebResource (jersey)
The image name you're passing to pull isn't valid. The format for an image name is name:tag. Do this instead:
docker.pull("10.###.###.143:5000/user/ipatest:latest");
Alternatively, you can ommit the :latest, since it is implied when no other tag is specified.
I have problem with API that I am creating. I am using Restlet for create API with json response. First of all I read documentation of restlet and I use simple model that they introduce here: http://restlet.org/discover/firststeps
My code is pretty much the same but some parts. My ServerResource is this:
public class CategoriesResource extends ServerResource {
#Get("json")
public StringRepresentation represent(Representation entity) {
dbOperations db = new dbOperations();
db.connect();
Series<Header> responseHeaders = (Series<Header>) getResponse().getAttributes().get("org.restlet.http.headers");
if (responseHeaders == null) {
responseHeaders = new Series(Header.class);
getResponse().getAttributes().put("org.restlet.http.headers", responseHeaders);
}
responseHeaders.add("Access-Control-Allow-Origin", "*");
return new StringRepresentation(db.getCategoriesJson(), MediaType.APPLICATION_JSON);
}
}
I creating json response with gson library in db operations I have lines:
CategoriesGson json = null;
json = new CategoriesGson(a, b, c, d);
// databases operations, most getting information from database
jsonList.add(gson.toJson(json));
return jsonList.toString();
Main problem is with CPU load when I press F5 after few times I get a lot of load on CPU. I was searching problem with that and I was able to do dump of thread is giving this usage.
Restlet-9860934" prio=10 tid=0x8828d000 nid=0x12ee runnable [0x87f89000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.valueOf(String.java:2854)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at org.restlet.engine.connector.Way.toString(Way.java:594)
at java.lang.String.valueOf(String.java:2854)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at org.restlet.engine.connector.Way.onSelected(Way.java:471)
at org.restlet.util.SelectionRegistration.onSelected(SelectionRegistration.java:325)
at org.restlet.engine.connector.Connection.onSelected(Connection.java:612)
- locked <0x937a7550> (a java.nio.HeapByteBuffer)
at org.restlet.util.SelectionRegistration.onSelected(SelectionRegistration.java:325)
at org.restlet.engine.connector.ConnectionController.onSelected(ConnectionController.java:219)
at org.restlet.engine.connector.ServerConnectionController.onSelected(ServerConnectionController.java:99)
at org.restlet.engine.connector.ConnectionController.selectKeys(ConnectionController.java:308)
at org.restlet.engine.connector.ConnectionController.doRun(ConnectionController.java:171)
at org.restlet.engine.connector.Controller.run(Controller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
This is restlet thread and I think this is it fault. I have not idea how I can fix that. I was searching for this issue and I found only issue 386 and 794 on github. If someone have some idea how I can fix that please let me know.
I was having exactly the same bug and I just figured out how to reproduce it:
public Restlet createInboundRoot() {
Router router = new Router(getContext());
ChallengeAuthenticator authenticator = new ChallengeAuthenticator(getContext(), ChallengeScheme.HTTP_BASIC, "cancel");
Verifier verifier = new Verifier() { // not even used
#Override
public int verify(Request arg0, Response arg1) {
return Verifier.RESULT_VALID;
}
};
authenticator.setVerifier(verifier);
router.attach("/test", HelloResource.class);
Authorizer authorizer = new Authorizer() { // authorizer only being used to delay request
#Override
protected boolean authorize(Request arg0, Response arg1) {
try {
Thread.sleep(5000); // request is delayed by 5000ms
} catch (InterruptedException e) {
e.printStackTrace();
}
return true;
}
};
authorizer.setNext(router);
authenticator.setNext(authorizer);
return authenticator;
}
So what I did was pretty simple. I delayed the request by 5s.
When I request a resource in my browser I cancel the request right away. Afterwards my CPU usage goes up to 100% and keeps at that level ...