I am using Eclipse Jetty HttpClient to send POST requests to a server, for load testing.
TL;DR: Is there a way to use a single HttpClient instance with multiple user credential sets to a single destination URL?
For this purpose, I need to log in to the server-under-test as separate users.
Even though HttpClient is thread safe, it does not appear to support this with a single instance, due to its shared authentication store.
The solution seems easy, just use one HttpClient per user, or per thread.
This works okay, except that HttpClient creates a number of threads (5 to 10 it seems) for each instance, and so my load test needs a very large heap or else it will start throwing OutOfMemory exceptions when trying to create new threads.
For example, in this very basic test, the first set of credentials is used for all subsequent POSTs:
public class Test
{
static class User
{
String username;
String password;
User(String username, String password)
{
this.username = username;
this.password = password;
}
}
public static void main(String[] args) throws Exception
{
SslContextFactory sslContextFactory = new SslContextFactory.Client();
HttpClient httpClient = new HttpClient(sslContextFactory);
httpClient.start();
List<User> users = new ArrayList<>();
users.add(new User("fry", "1234"));
users.add(new User("leela", "2345"));
users.add(new User("zoidberg", "3456"));
URI uri = new URI("http://localhost:8080/myapi");
for (User user : users)
{
AuthenticationStore auth = httpClient.getAuthenticationStore();
auth.addAuthentication(new DigestAuthentication(uri, DigestAuthentication.ANY_REALM, user.username, user.password));
Request request = httpClient.newRequest(uri);
request.method("POST");
ContentResponse result = request.send();
System.out.println(result.getStatus());
}
}
}
Now, I realize in this contrived test that I can call httpClient.getAuthenticationStore().clearAuthenticationResults() and httpClient.getAuthenticationStore().clearAuthentications(); between loops, however that does not work for my actual testing, where I have multiple threads posting at the same time.
Am I stuck using a separate HttpClient instance for each user?
Thanks for any ideas!
What you need can be done by "preempting" the authentication headers for every request, as explained in the documentation.
This is how you would do it:
// Single HttpClient instance.
HttpClient httpClient = new HttpClient();
// The server URI.
URI uri = URI.create("http://example.com/secure");
// The authentication credential for each user.
Authentication.Result authn1 = new BasicAuthentication.BasicResult(uri, "user1", "password1");
Authentication.Result authn2 = new BasicAuthentication.BasicResult(uri, "user2", "password2");
// Create a request instance.
Request request1 = httpClient.newRequest(uri);
// Add the authorization headers for user1.
authn1.apply(request1);
request1.send();
Request request2 = httpClient.newRequest(uri);
// Add the authorization headers for user2.
authn2.apply(request2);
request2.send();
Sending the requests does not need to be sequential or using the blocking APIs like in the simple example above.
You can do it from a for loop, and pick a user (and its correspondent authorization) randomly, for example, and use the asynchronous APIs for better performance.
Related
I'm struggling with invoking GCP cloud functions via REST API using Java.
The steps that I've performed to do it were:
create a service account with role "Cloud Functions Invoker"
download JSON key file for the newly created service account
in my code, obtain an access token using the following method:
private String getAuthToken() {
File credentialsPath = new File(PATH_TO_JSON_KEY_FILE);
GoogleCredentials credentials;
try (FileInputStream serviceAccountStream = new FileInputStream(credentialsPath)) {
credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
return credentials
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"))
.refreshAccessToken()
.getTokenValue();
} catch (IOException e) {
throw new RuntimeException("Action could not be performed");
}
}
perform a REST call, using the created token:
public <Payload, Response> ResponseEntity<Response> callCloudFunction(
String endpoint,
Payload payload,
Class<Response> klazz
) {
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
headers.setContentType(MediaType.APPLICATION_JSON);
String url = gCloudUrl + endpoint;
String token = getAuthToken();
String payloadString = null;
if (payload != null) {
try {
ObjectMapper objectMapper = new ObjectMapper();
payloadString = objectMapper.writeValueAsString(payload);
} catch (JsonProcessingException e) {
System.out.println(e.getMessage());
throw new RuntimeException("Could not perform action");
}
}
headers.add("Authorization", String.format("Bearer %s", token));
HttpEntity<String> entity = new HttpEntity<>(payloadString, headers);
return restTemplate.exchange(url, HttpMethod.POST, entity, klazz);
}
The implementation looks fine, but in response I'm getting 401 Unauthorized.
Unfortunately, GCP documentation is not really helpful. I think I've searched through all the possible places.
First of all, agree, it's not clear...
Then, you have to know (and it's not clear again) that you need an access token to call Google Cloud API, but and identity token to call IAP (on App Engine for example) or private Cloud Function and Cloud Run. And this identity token need to be signed by Google.
And, as mentioned in the code, you need to have a service account on your computer, but I recommend you to avoid this on GCP, it's not required if you use default authentication (see my code, on your computer set the GOOGLE_APPLICATION_CREDENTIALS env var that points to the service account key file). The best way is to not use service account key file on your computer also, but it's not yet possible (that is a security issue IMO, and I'm discussing with Google on this...)
Anyway, here a code snippet which works in Java (nowhere in the documentation...)
String myUri = "https://path/to/url";
// You can use here your service account key file. But, on GCP you don't require a service account key file.
// However, on your computer, you require one because you need and identity token and you can generate it with your user account (long story... I'm still in discussion with Google about this point...)
Credentials credentials = GoogleCredentials.getApplicationDefault().createScoped("https://www.googleapis.com/auth/cloud-platform");
IdTokenCredentials idTokenCredentials = IdTokenCredentials.newBuilder()
.setIdTokenProvider((IdTokenProvider) credentials)
.setTargetAudience(myUri).build();
HttpRequestFactory factory = new NetHttpTransport().createRequestFactory(new HttpCredentialsAdapter(idTokenCredentials));
HttpRequest request = factory.buildGetRequest(new GenericUrl(myUri));
HttpResponse httpResponse = request.execute();
System.out.println(CharStreams.toString(new InputStreamReader(httpResponse.getContent(), Charsets.UTF_8)));
NOTE If you want to continue to use RestTemplate object and set manually your token, you can generate it like this
String token = ((IdTokenProvider) credentials).idTokenWithAudience(myUri, Collections.EMPTY_LIST).getTokenValue();
System.out.println(token);
I need a little help with this, I've used HTTP requests before but through JavaScript when I was using AJAX, it looks similar but I'm not entirely sure how to do all of it properly. I've seen some people's codes and stuff, but I'm not sure about two things. First, I want to send an int through the POST in order to identify which query should be called and what data should be sent back through the response. Second, I'm not sure about the URL, do I need a domain? I just need to connect to a server on my machine which I'm using with XAMPP. I'm not entirely sure how to do this? Do I just place a URL with my machine's IP with a port or something?
So here's what I've attempted:
public class Request {
private static Request instance;
private static String URL;
private String requestResult;
private String error;
private Request(){
this.URL = "http://IPAddress or Server Address/Android Webservice/webservice.php"; /* not sure what I should use here, I'm using an apache server */
this.error = new String();
this.requestResult = new String();
}
public static Request getRequest(){
if(instance == null){
instance = new Request();
}
return instance;
}
public void sendHttpPostRequest(String option){
HttpClient client = new DefaultHttpClient();
HttpPost postRequest = new HttpPost(URL);
HttpResponse serverResponse;
String jsonToSend = "message={identifer:" + option + "}";
NameValuePair parameter = new BasicNameValuePair("data", jsonToSend);
try {
StringEntity entity = new StringEntity(jsonToSend);
postRequest.addHeader("content-type", "application/x-www-form-urlencoded");
postRequest.setEntity(entity);
serverResponse = client.execute(postRequest);
} catch (Exception e){
}
}
public String getURL() {
return this.URL;
}
public String getResult(){
return this.requestResult;
}
}
I've got it in a class, because I want to be able to call it anywhere, plus I need to call it in multiple views in my android application. I would say the most important part in here is the `sendHttpPostRequest. However I'm not sure how to go about using that response to do anything. Normally with AJAX I'd just get the response text and go from there, but that doesn't work here. I've seen various examples but I'm having a hard time getting them, and some of them do a lot of stuff that I'm not sure I need.
For instance this is my PHP file:
<?php
$obj = json_decode($_POST['message']);
$obj = json_decode($dataReceived,true);
$data = array("data","some other data","the last piece of data");
json_encode('data'=>$data);
echo $data;
?>
For now it looks like this because this is a simple test, but what the end goal should be is to return an an associative array which would be the result of a query from my database. I know how to do that easily, but I'm not sure how to send the request to get this file to return that stuff. I don't know how to parse a JSON within Java, especially an associative array and I'm also not sure how my server should be configured to allow connections from android devices.
So what exactly needs to go in the URL to connect to my PC? How do I parse JSON response and then convert it into data that I can place in my android application? How do I go about moving anywhere with this device, and retrieving this data from a location far from my server?
I'm hoping that this doesn't require payment of any sort, I mean the server is on my computer after all.
Just password protected solr on Jetty server. I am able to read/access data from solr using solrj :->
HttpSolrServer solr = new HttpSolrServer("http://111.111.111:8983/solr/mysolr");
HttpClientUtil.setBasicAuth((DefaultHttpClient) solr.getHttpClient(), "admin", "akshaybhatt");
but it gives me I/O Exception as below. There are other examples on SO about authentication but I have no idea how do I use authentication in Solrj. The below error comes only when I try to update a record (and possibly add a record, not tested yet)
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://111.111.111.138:8983/solr/mysolrcore
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:507)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:118)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at UpdateSolrTesting.AddToSolr(UpdateSolrTesting.java:228)
at UpdateSolrTesting.performaction(UpdateSolrTesting.java:141)
at UpdateSolrTesting.main(UpdateSolrTesting.java:101)
Caused by: org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
... 7 more
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:660)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
... 11 more
It is happening only because authentication parameter is not being sent while doing POST or DELETE calls, so so solution is you need to fix that in your http client
I am using solr 6.2.0 and its corresponding java client
So i created a new SolrHttp client which looks like below
public class TEHttpSolrClient extends HttpSolrClient {
private static final String UTF_8 = StandardCharsets.UTF_8.name();
public TEHttpSolrClient(String baseURL) {
super(baseURL);
}
#Override
public NamedList<Object> request(final SolrRequest request, String collection) throws SolrServerException, IOException {
ResponseParser responseParser = request.getResponseParser();
if (responseParser == null) {
responseParser = parser;
}
return request(request, responseParser, collection);
}
public NamedList<Object> request(final SolrRequest request, final ResponseParser processor, String collection)
throws SolrServerException, IOException {
HttpRequestBase method = createMethod(request, collection);
String userPass = "<username>:<password>";
String encoded = Base64.byteArrayToBase64(userPass.getBytes(UTF_8));
// below line will make sure that it sends authorization token every time in all your requests
method.setHeader(new BasicHeader("Authorization", "Basic " + encoded));
return executeMethod(method, processor);
}
}
Also to call the client you should call it like below
private static SolrClient solr = new TEHttpSolrClient.Builder("<solr core url>").build();
What you need is called "preemptive authentication". This tells the http client to authenticate on the first call to the url. The default behaviour is to send two request, when basic authentication is used. This might fail, as in your case, when the entity in the http call is not reusable.
Luckyly solr allready has a build in way to enable preemptive authentication by using a different client building factory for SolrHttpClientBuilder.
String userName = "someUserName";
String password = "secretPassword";
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_BASIC_AUTH_USER, userName);
params.set(HttpClientUtil.PROP_BASIC_AUTH_PASS, password);
// set the params for authentication here
PreemptiveBasicAuthClientBuilderFactory.setDefaultSolrParams(params);
PreemptiveBasicAuthClientBuilderFactory preemptiveBasicAuthClientBuilderFactory = new PreemptiveBasicAuthClientBuilderFactory();
// create a new client builder from the preemptive client builder factory
SolrHttpClientBuilder httpClientBuilder = preemptiveBasicAuthClientBuilderFactory
.getHttpClientBuilder(Optional.empty());
// set the client builder to be used by the clientUtil
HttpClientUtil.setHttpClientBuilder(httpClientBuilder);
// the params need to be passed here too
CloseableHttpClient httpAuthClient = HttpClientUtil.createClient(params);
// now build the solr client with the special http client
Builder solrClientBuilder = new HttpSolrClient.Builder(solrClientConfig.getSolrUrl()).withHttpClient(httpAuthClient);
// create solr client
SolrClient client = solrClientBuilder.build();
Remember not to set the authorization params at the request level, otherwise the preemptive auth won't work.
Similar question asked already, you can look at the below links to get some idea.
Solr - instantiate HttpSolrServer with Httpclient
Solr Change CommonsHttpSolrServer To HttpSolrServer
I need to transport certain data from one request to another for the oauth workflow.
#RequestMapping(value = "/connect/twitter", method = RequestMethod.POST)
public RedirectView connectTwitter(HttpServletRequest request,
Model model) {
TwitterConnectionFactory connectionFactory = new TwitterConnectionFactory(
environment.getProperty("spring.social.twitter.app-id"),
environment.getProperty("spring.social.twitter.app-secret"));
OAuth1Operations oauthOperations = connectionFactory.getOAuthOperations();
OAuthToken requestToken = oauthOperations.fetchRequestToken(request.getRequestURL().toString(), null);
String authorizeUrl = oauthOperations.buildAuthorizeUrl(requestToken.getValue(), OAuth1Parameters.NONE);
//need requestToken in the next process
return new RedirectView(authorizeUrl);
}
#RequestMapping(value = "/connect/twitter", method = RequestMethod.GET)
#ResponseBody
public String verifyTwitter(#RequestParam("oauth_token") String oauthToken,
#RequestParam("oauth_verifier") String oauthVerifier,
OAuthToken requestToken /*need requestToken from last request*/) {
TwitterConnectionFactory connectionFactory = new TwitterConnectionFactory(
environment.getProperty("spring.social.twitter.app-id"),
environment.getProperty("spring.social.twitter.app-secret"));
OAuth1Operations oauthOperations = connectionFactory.getOAuthOperations();
OAuthToken accessToken = oauthOperations.exchangeForAccessToken(new AuthorizedRequestToken(requestToken, oauthVerifier), null);
Connection<Twitter> twitterConnection = connectionFactory.createConnection(accessToken);
return "asd";
}
the requestToken from the frist request has to be available in the next request. how to handle it?
Well, one way to do it is to store it in "session". I say put quotes around that because I don't necessarily mean servlet session (which may or may not work across multiple nodes, depending on your server setup). It could be anything that performs the function of session, such as (perhaps) a Redis key-value store. Of course, once you fetch it from "session", you'll also want to clean it out.
Spring MVC supports flash attributes directly for this purpose. See http://docs.spring.io/spring/docs/4.0.6.RELEASE/spring-framework-reference/htmlsingle/#mvc-flash-attributes.
Also, it strikes me that you're writing your own controller to do the OAuth dance with Twitter, but Spring Social's ConnectController already exists for that purpose. See https://github.com/spring-projects/spring-social-samples/tree/master/spring-social-showcase for an example of how ConnectController is used.
I'm trying to write a simple smoke test for a web application.
The application normally uses form based authentication, but accepts basic auth as well, but since the default is form based authentication, it never sends an authentication required, but instead just sends the login form.
In the test I try to send the basic auth header using
WebClient webClient = new WebClient();
DefaultCredentialsProvider creds = new DefaultCredentialsProvider();
// Set some example credentials
creds.addCredentials("usr", "pwd");
// And now add the provider to the webClient instance
webClient.setCredentialsProvider(creds);
webClient.getPage("<some url>")
I also tried stuffing the credentials in a WebRequest object and passing that to the webClient.getPage method.
But on the server I don't get an authentication header. I suspect the WebClient only sends the authentication header if it get explicitly asked for it by the server, which never happens.
So the question is how can I make the WebClient send the Authentication header on every request, including the first one?
This might help:
WebClient.addRequestHeader(String name, String value)
More specific one can create an authentication header like this
private static void setCredentials(WebClient webClient)
{
String username = "user";
String password = "password";
String base64encodedUsernameAndPassword = base64Encode(username + ":" + password);
webClient.addRequestHeader("Authorization", "Basic " + base64encodedUsernameAndPassword);
}
private static String base64Encode(String stringToEncode)
{
return DatatypeConverter.printBase64Binary(stringToEncode.getBytes());
}