We Keep Coding

upwork-api return 503 ioexception

I created app for getting info from upwork.com. I use java lib and Upwork OAuth 1.0. The problem is local request to API works fine, but when I do deploy to Google Cloud, my code does not work. I get ({"error":{"code":"503","message":"Exception: IOException"}}).
I create UpworkAuthClient for return OAuthClient and next it is used for requests in JobClient.
run() {
UpworkAuthClient upworkClient = new UpworkAuthClient();
upworkClient.setTokenWithSecret("USER TOKEN", "USER SECRET");
OAuthClient client = upworkClient.getOAuthClient();
//set query
JobQuery jobQuery = new JobQuery();
jobQuery.setQuery("query");
List<JobQuery> jobQueries = new ArrayList<>();
jobQueries.add(jobQuery);
// Get request of job
JobClient jobClient = new JobClient(client, jobQuery);
List<Job> result = jobClient.getJob();
}
public class UpworkAuthClient {
public static final String CONSUMERKEY = "UPWORK KEY";
public static final String CONSUMERSECRET = "UPWORK SECRET";
public static final String OAYTРCALLBACK = "https://my-app.com/main";
OAuthClient client ;
public UpworkAuthClient() {
Properties keys = new Properties();
keys.setProperty("consumerKey", CONSUMERKEY);
keys.setProperty("consumerSecret", CONSUMERSECRET);
Config config = new Config(keys);
client = new OAuthClient(config);
}
public void setTokenWithSecret (String token, String secret){
client.setTokenWithSecret(token, secret);
}
public OAuthClient getOAuthClient() {
return client;
}
public String getAuthorizationUrl() {
return this.client.getAuthorizationUrl(OAYTРCALLBACK);
}
}
public class JobClient {
private JobQuery jobQuery;
private Search jobs;
public JobClient(OAuthClient oAuthClient, JobQuery jobQuery) {
jobs = new Search(oAuthClient);
this.jobQuery = jobQuery;
}
public List<Job> getJob() throws JSONException {
JSONObject job = jobs.find(jobQuery.getQueryParam());
jobList = parseResponse(job);
return jobList;
}
}
Local dev server works fine, I get resilts on local machine, but in Cloud not.
I will be glad to any ideas, thanks!

{"error":{"code":"503","message":"Exception: IOException"}}
doesn't seem like a response return by Upwork API. Could you please provide the full response including the returned headers? So, we will take a more precise look into it.

How to pass data from vaadin webapp to C# GUI app

I have webapp in Vaadin Framework 8. I have Windows GUI app in C#.
The gui app is using WebBrowser component to display webapp. WebBrowser component is internally using IE11 core through ActiveX. I can successfully load and display the webapp in the gui app browser component.
I need to pass data from webapp to the gui app.
The webapp has many rows loaded on server side, only few are displayed in grid. I want to pass all data from webapp to gui app in some format (csv or json).
I have tryed some approaches, but I wasn't successfull.
[Approach 1]
Webapp: attach downloadable resource (csv) to Link with predefined id using FileDownloader. Download by user mouse click works fine, file save dialog pops up and data are downloaded successfully.
Link link = new Link("Data");
link.setId("myId");
StreamResource resource = getMyResource(data);
FileDownloader downloader = new FileDownloader(resource);
downloader.extend(link);
Page.getCurrent().getJavaScript().addFunction("test", new JavaScriptFunction() {
#Override
public void call(JsonArray arguments) {
Page.getCurrent().getJavaScript()
.execute("document.getElementById('myId').click()");
}
});
Gui app: raise onClick event on link and capture WebBrowser.FileDownload event, capture WebBrowser.Navigate event.
I have failed to raise onClick event from C# using:
HtmlElement el = webBrowser.Document.GetElementById("myId");
el.RaiseEvent("onClick");
el.InvokeMember("click");
webBrowser.Document.InvokeScript("document.getElementById('myId').click();", null);
webBrowser.Document.InvokeScript("test", null);
Result:
WebBrowser.FileDownload event doesn't work (is fired but can't capture url nor data), capture WebBrowser.Navigate event works partialy (can see resource url, but can't download data using byte[] b = new WebClient().DownloadData(e.Url);).
[Approach 2]
Similar to approach 1. I tryed to get resource url, put the direct url to Link and download the resource in c# using direct link. I can construct the same resource url as is used by browser to download data when user clicks the link.
Extended file downloader that keeps resource, key and connector:
public class ExtendedFileDownloader extends FileDownloader {
private String myKey;
private Resource myResource;
private ClientConnector myConnector;
public ExtendedFileDownloader(StreamResource resource, ClientConnector connector) {
super(resource);
myConnector = connector;
}
#Override
protected void setResource(String key, Resource resource) {
super.setResource(key, resource);
myKey = key;
myResource = resource;
}
public String getResourceUrl() {
ResourceReference ref =
ResourceReference.create(
myResource,
(myConnector != null) ? myConnector : this,
myKey);
String url = ref.getURL();
return url;
}
}
In view:
// fix app://path... urls to /<base-path>/path urls
private String fixResourceReferenceUrl(String resourceReferenceUrl) {
String resourceReferencePath = resourceReferenceUrl.replace("app://", "");
String uiBaseUrl = ui.getUiRootPath();
String fixedUrl = uiBaseUrl + "/" + resourceReferencePath;
return fixedUrl;
}
Link link2 = new Link("Data2");
link2.setId("myId2");
StreamResource resource = getMyResource(data);
ExtendedFileDownloader downloader = new ExtendedFileDownloader(resource, this);
String fixedResourceUrl = fixResourceReferenceUrl(downloader.getResourceUrl());
link2.setResource(new ExternalResource(fixedResourceUrl));
Result:
The data cannot be downloaded using this link, server error 410 or NotFound errors.
Any Ideas ? Any other approaches to try ?

I have finally solved the problem. The solution is very close to approach 2.
The resource url is passed in element with custom attribute. C# WebClient needs to set cookies from WebBrowser and Referer HTTP headers. The data can be successfully downloaded by C# app.
Element attribute in vaadin webapp can be set using Vaadin-addon Attributes.
Cookies in C# app can be retrieved using this solution.
// Fix resource urls begining with app://
public String fixResourceReferenceUrl(String resourceReferenceUrl) {
try {
String uiRootPath = UI.getCurrent().getUiRootPath();
URI location = Page.getCurrent().getLocation();
String appLocation = new URIBuilder()
.setScheme(location.getScheme())
.setHost(location.getHost())
.setPort(location.getPort())
.setPath(uiRootPath)
.build()
.toString();
String resourceReferencePath = resourceReferenceUrl.replace("app://", "");
String fixedUrl = appLocation + "/" + resourceReferencePath;
return fixedUrl;
}
catch (Exception e) {
return null;
}
}
In view (using ExtendedFileDownloader from above):
Link link = new Link("Data");
link.setId("myId");
StreamResource resource = getMyResource(data);
ExtendedFileDownloader downloader = new ExtendedFileDownloader(resource);
downloader.extend(link);
Attribute attr = new Attribute("x-my-data", fixResourceReferenceUrl(downloader.getResourceUrl()));
attr.extend(link);
link.setVisible(true);
In C# app:
[DllImport("wininet.dll", SetLastError = true)]
public static extern bool InternetGetCookieEx(
string url,
string cookieName,
StringBuilder cookieData,
ref int size,
Int32 dwFlags,
IntPtr lpReserved);
private const Int32 InternetCookieHttponly = 0x2000;
public static String GetUriCookies(String uri)
{
// Determine the size of the cookie
int datasize = 8192 * 16;
StringBuilder cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(uri, null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero))
{
if (datasize < 0)
return null;
// Allocate stringbuilder large enough to hold the cookie
cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(
uri,
null, cookieData,
ref datasize,
InternetCookieHttponly,
IntPtr.Zero))
return null;
}
return cookieData.ToString();
}
private void button_Click(object sender, EventArgs e)
{
HtmlElement el = webBrowser.Document.GetElementById("myId");
String url = el.GetAttribute("x-my-data");
String cookies = GetUriCookies(url);
WebClient wc = new WebClient();
wc.Headers.Add("Cookie", cookies);
wc.Headers.Add("Referer", WEB_APP_URL); // url of webapp base path, http://myhost/MyUI
byte[] data = wc.DownloadData(url);
}

How can I append a query parameter to an existing URL?

I'd like to append key-value pair as a query parameter to an existing URL. While I could do this by checking for the existence of whether the URL has a query part or a fragment part and doing the append by jumping though a bunch of if-clauses but I was wondering if there was clean way if doing this through the Apache Commons libraries or something equivalent.
http://example.com would be http://example.com?name=John
http://example.com#fragment would be http://example.com?name=John#fragment
http://example.com?email=john.doe#email.com would be http://example.com?email=john.doe#email.com&name=John
http://example.com?email=john.doe#email.com#fragment would be http://example.com?email=john.doe#email.com&name=John#fragment
I've run this scenario many times before and I'd like to do this without breaking the URL in any way.

There are plenty of libraries that can help you with URI building (don't reinvent the wheel). Here are three to get you started:
Java EE 7
import javax.ws.rs.core.UriBuilder;
...
return UriBuilder.fromUri(url).queryParam(key, value).build();
org.apache.httpcomponents:httpclient:4.5.2
import org.apache.http.client.utils.URIBuilder;
...
return new URIBuilder(url).addParameter(key, value).build();
org.springframework:spring-web:4.2.5.RELEASE
import org.springframework.web.util.UriComponentsBuilder;
...
return UriComponentsBuilder.fromUriString(url).queryParam(key, value).build().toUri();
See also: GIST > URI Builder Tests

This can be done by using the java.net.URI class to construct a new instance using the parts from an existing one, this should ensure it conforms to URI syntax.
The query part will either be null or an existing string, so you can decide to append another parameter with & or start a new query.
public class StackOverflow26177749 {
public static URI appendUri(String uri, String appendQuery) throws URISyntaxException {
URI oldUri = new URI(uri);
String newQuery = oldUri.getQuery();
if (newQuery == null) {
newQuery = appendQuery;
} else {
newQuery += "&" + appendQuery;
}
return new URI(oldUri.getScheme(), oldUri.getAuthority(),
oldUri.getPath(), newQuery, oldUri.getFragment());
}
public static void main(String[] args) throws Exception {
System.out.println(appendUri("http://example.com", "name=John"));
System.out.println(appendUri("http://example.com#fragment", "name=John"));
System.out.println(appendUri("http://example.com?email=john.doe#email.com", "name=John"));
System.out.println(appendUri("http://example.com?email=john.doe#email.com#fragment", "name=John"));
}
}
Shorter alternative
public static URI appendUri(String uri, String appendQuery) throws URISyntaxException {
URI oldUri = new URI(uri);
return new URI(oldUri.getScheme(), oldUri.getAuthority(), oldUri.getPath(),
oldUri.getQuery() == null ? appendQuery : oldUri.getQuery() + "&" + appendQuery, oldUri.getFragment());
}
Output
http://example.com?name=John
http://example.com?name=John#fragment
http://example.com?email=john.doe#email.com&name=John
http://example.com?email=john.doe#email.com&name=John#fragment

For android, Use:
https://developer.android.com/reference/android/net/Uri#buildUpon()
URI oldUri = new URI(uri);
Uri.Builder builder = oldUri.buildUpon();
builder.appendQueryParameter("newParameter", "dummyvalue");
Uri newUri = builder.build();

Use the URI class.
Create a new URI with your existing String to "break it up" to parts, and instantiate another one to assemble the modified url:
URI u = new URI("http://example.com?email=john#email.com&name=John#fragment");
// Modify the query: append your new parameter
StringBuilder sb = new StringBuilder(u.getQuery() == null ? "" : u.getQuery());
if (sb.length() > 0)
sb.append('&');
sb.append(URLEncoder.encode("paramName", "UTF-8"));
sb.append('=');
sb.append(URLEncoder.encode("paramValue", "UTF-8"));
// Build the new url with the modified query:
URI u2 = new URI(u.getScheme(), u.getAuthority(), u.getPath(),
sb.toString(), u.getFragment());

I suggest an improvement of the Adam's answer accepting HashMap as parameter
/**
* Append parameters to given url
* #param url
* #param parameters
* #return new String url with given parameters
* #throws URISyntaxException
*/
public static String appendToUrl(String url, HashMap<String, String> parameters) throws URISyntaxException
{
URI uri = new URI(url);
String query = uri.getQuery();
StringBuilder builder = new StringBuilder();
if (query != null)
builder.append(query);
for (Map.Entry<String, String> entry: parameters.entrySet())
{
String keyValueParam = entry.getKey() + "=" + entry.getValue();
if (!builder.toString().isEmpty())
builder.append("&");
builder.append(keyValueParam);
}
URI newUri = new URI(uri.getScheme(), uri.getAuthority(), uri.getPath(), builder.toString(), uri.getFragment());
return newUri.toString();
}

Use this function to append a new parameter to your existing URI
private Uri appendUriParameter(Uri uri, String key, String newValue) {
final Set<String> params = uri.getQueryParameterNames();
final Uri.Builder newUri = uri.buildUpon().clearQuery();
for (String param : params) {
newUri.appendQueryParameter(param, uri.getQueryParameter(param));
}
newUri.appendQueryParameter(key, newValue);
return newUri.build();
}

Kotlin & clean, so you don't have to refactor before code review:
private fun addQueryParameters(url: String?): String? {
val uri = URI(url)
val queryParams = StringBuilder(uri.query.orEmpty())
if (queryParams.isNotEmpty())
queryParams.append('&')
queryParams.append(URLEncoder.encode("$QUERY_PARAM=$param", Xml.Encoding.UTF_8.name))
return URI(uri.scheme, uri.authority, uri.path, queryParams.toString(), uri.fragment).toString()
}

An update to Adam's answer considering tryp's answer too. Don't have to instantiate a String in the loop.
public static URI appendUri(String uri, Map<String, String> parameters) throws URISyntaxException {
URI oldUri = new URI(uri);
StringBuilder queries = new StringBuilder();
for(Map.Entry<String, String> query: parameters.entrySet()) {
queries.append( "&" + query.getKey()+"="+query.getValue());
}
String newQuery = oldUri.getQuery();
if (newQuery == null) {
newQuery = queries.substring(1);
} else {
newQuery += queries.toString();
}
URI newUri = new URI(oldUri.getScheme(), oldUri.getAuthority(),
oldUri.getPath(), newQuery, oldUri.getFragment());
return newUri;
}

Is it possible to get the absolute request url without path parameters

I am trying to get the request url without values of path parameters into it.
Consider my complete url is
URl: http://localhost:8080/aaa/mock/abcd/1234/true
Path parameters: abcd, true
Output needed: /aaa/mock/abcd
My web service method looks like this.
#Path(value = "/aaa/mock")
#Component
public class MockService
{
private static Log log = LogFactory.getLog(MockService.class);
//address
#GET
#Path(value = "/{mockrequest}/{status}")
#Produces(MediaType.JSON)
public String mockEngagement(#Context ContainerRequestContext request,#PathParam("mockrequest") String mockrequest,#PathParam("status") String status )
{
log.info("The mock url is"+request.getUriInfo().getRequestUri());
log.info("The mock url is"+request.getUriInfo().getAbsolutePath());
log.info("The mock url is"+request.getUriInfo().getBaseUri());
log.info("The mock url is"+request.getUriInfo().getMatchedURIs());
**//Out put needed /aaa/mock/abcd**
return "ajaja";
}
}
None of the above calls return the required info.
I am thinking if there is a generic process to get the desired output irrespective of number of path parameters.
Any such methods.

Try UriInfo#getPath(), UriInfo#getPath(boolean), or UriInfo#getPathSegments(). The boolean argument is whether the path should be encoded or not.
https://jersey.java.net/apidocs/2.3.1/jersey/index.html
You could also get the absolute path and the base path and then use URI#relativize(URI).

Try this:
request.getUriInfo().getPathSegments().get(0).getPath()

public void filter(ContainerRequestContext context) throws IOException {
Message message = PhaseInterceptorChain.getCurrentMessage();
Set<Map.Entry<String, Object>> o = (Set<Map.Entry<String, Object>>)message.entrySet();
for (Map.Entry<String, Object> oo : o) {
String key = oo.getKey();
Object val = oo.getValue();
// Thises two properties gives the path of web service
//path_to_match_slash
//org.apache.cxf.request.uri
if(key.equals("path_to_match_slash"))
{ String v = (String)val;
System.out.println (key);
System.out.println (v);
}
if(key.equals("org.apache.cxf.request.uri"))
{ String v = (String)val;
System.out.println (key);
System.out.println (v);
}
}
}
this code could work only for apache cxf rest
we can found path_to_match_slash , org.apache.cxf.request.uri properties in the ContainerRequestContext

Wicket image component not found on page

I get the following problem when trying to display a list of items. For each item, I have to display an image which is dynamically loaded via a Wicket WebResource. The items are loaded step by step — 50 at a time — upon user scrolling, using an Ajax scroll.
[ERROR] 2011-04-19 09:58:18,000 btpool0-1 org.apache.wicket.RequestCycle.logRuntimeException (host=, request=, site=):
org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail not found on page com.webapp.document.pages.DocumentListPage[id = 1]
listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
org.apache.wicket.protocol.http.request.InvalidUrlException: org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail
not found on page com.webapp.document.pages.DocumentListPage[id = 1] listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
at org.apache.wicket.protocol.http.WebRequestCycleProcessor.resolve(WebRequestCycleProcessor.java:262)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1310)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1428)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:479)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doGet$6()
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816$$FastClassByGuice$$6d42bf5d.invoke()
at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.freiheit.monitoring.PerformanceMonitoringMethodInterceptor.invoke(PerformanceMonitoringMethodInterceptor.java:115)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.doGet()
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:312)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doFilter$4()
How can this problem be solved?
Here is the part of the code responsible for adding the image:
previewLink.add(createThumbnailSmall("imageThumbnail", documentModel));
in
createThumbnailSmall(final String id, final IModel<BaseDocument> documentModel) {
// thumbnailResource is an object that contains the path of the image
if (thumbnailResource != null) {
final WebResource resource = getWebResource(thumbnailResource);
final Image image = new Image(id, resource);
return image;
}
return new InvisibleContainer(id);
}
WebResource getWebResource(final DocumentResource documentResource) {
return new WebResource() {
private static final long serialVersionUID = 1L;
#Override
public IResourceStream getResourceStream() {
return new BaseStreamResource(documentResource);
}
};
}
where BaseStreamResource is the following:
public class BaseStreamResource extends AbstractResourceStream {
private InputStream _fileInputStream = null;
private DocumentResource _resource = null;
public BaseStreamResource(final DocumentResource documentResource) {
_resource = documentResource;
}
#Override
public InputStream getInputStream() throws ResourceStreamNotFoundException {
if (_fileInputStream == null) {
try {
if (_resource == null) {
throw new ResourceStreamNotFoundException("Resource was null");
}
_fileInputStream = _resource.getFileInputStream();
} catch (final ResourceNotAvailableException ex) {
throw new ResourceStreamNotFoundException(ex);
}
}
return _fileInputStream;
}
In HTML:
<a wicket:id="linkToPreview" href="#">
<img wicket:id="imageThumbnail" alt="Attachment"></img></a>

The code added hasn't really added any clues for me, but maybe I can help narrow it down a bit anyway.
The stacktrace includes a reference to com.webapp.document.pages.DocumentListPage, which is likely calling some of the code you've posted. The error indicates a bad url, so debugging into that class, adding debug prints, and looking at the values of any field containing a url might be worthwhile.
It might even help to modify the code in DocumentListPage (maybe temporarily for debugging) to catch org.apache.wicket.protocol.http.request.InvalidUrlException and adding debugging prints specifically when the exception is caught.
This isn't really an answer, but it's too big for a comment, and maybe it'll help you get closer to an answer.

The following solution solved the problem:
- extend WebResource class
- add extended class as a resource to application shared resources
Ex:
public class MyWebResource extends WebResource {
final ValueMap map = new ValueMap();
#Override
public IResourceStream getResourceStream() {
String fileName = getFileName();
File file = new File(basePath, fileName);
if (!file.exists()) {
LOG.error("File does not exist: " + file);
throw new IllegalStateException("File does not exist: " + file);
}
return new FileResourceStream(file);
}
public final void addResource() {
Application.get().getSharedResources().add(getClass().getName(), this);
}
protected String getFileName() {
return getParameters().getString("id");
}
public final String urlFor(final String fileName) {
final ResourceReference resourceReference = new ResourceReference(getClass().getName());
final String encodedValue = WicketURLEncoder.QUERY_INSTANCE.encode(fileName);
map.add("id", encodedValue);
final CharSequence result = RequestCycle.get().urlFor(resourceReference, map);
if (result == null) {
throw new IllegalStateException("The resource was not added! "
+ "In your Application class add the following line:"
+ "MyConcreteResource.INSTANCE.addResource()");
}
String absoluteUrl = RequestUtils.toAbsolutePath(result.toString());
return absoluteUrl;
}
}
In Application class, in init(), I have added MyWebResource to shared resources:
public void init() {
...
new MyWebResource().addResource();
...
}