We Keep Coding

Reading parquet files from S3 with custom endpoint in Java

I'm trying to find out the best way to read parquet data from S3 storage.
First approach:
BasicSessionCredentials cred = new BasicSessionCredentials(key,secret, "");
AmazonS3 client = AmazonS3ClientBuilder
.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("custom_endpoint", region))
.withCredentials(new AWSStaticCredentialsProvider(cred))
.build();
GetObjectRequest req = new GetObjectRequest("bucket_name", "relative_path", "");
S3Object obj = client.getObject(req);
S3ObjectInputStream cont = obj.getObjectContent();
This way I'm able to read the object, but I couldn't find the way of reading parquet data from InputStream
Second approach:
String SCHEMA_TEMPLATE = "{" +
"\"type\": \"record\",\n" +
" \"name\": \"schema\",\n" +
" \"fields\": [\n" +
" {\"name\": \"timeStamp\", \"type\": \"string\"},\n" +
" {\"name\": \"temperature\", \"type\": \"double\"},\n" +
" {\"name\": \"pressure\", \"type\": \"double\"}\n" +
" ]" +
"}";
String PATH_SCHEMA = "s3a";
Path internalPath = new Path(PATH_SCHEMA, bucketName, folderName);
Schema schema = new Schema.Parser().parse(SCHEMA_TEMPLATE);
Configuration configuration = new Configuration();
configuration.set("fs.s3a.access.key", "key");
configuration.set("fs.s3a.secret.key", "secret");
configuration.set("fs.s3a.endpoint", "custom_endpoint");
AvroReadSupport.setRequestedProjection(configuration, schema);
ParquetReader<GenericRecord> = AvroParquetReader.GenericRecord>builder(internalPath).withConf(configuration).build();
GenericRecord genericRecord = parquetReader.read();
while(genericRecord != null) {
Map<String, String> valuesMap = new HashMap<>();
genericRecord.getSchema().getFields().forEach(field -> valuesMap.put(field.name(), genericRecord.get(field.name()).toString()));
genericRecord = parquetReader.read();
}
But for the second case I'm not able to read data and getting SocketTimeoutException.
Help me to find the right approach
Thanks

Constant 401 error with OAuth1 (MCM API)

I've been trying to establish a connection with an API for more than a week now, to no avail. (Magic Card Market's, authentification documentation here and there). I'm supposed to receive a XML file.
I have what MCM call a "widget" access to their API, meaning that I don't have nor need a oauth_token (it's supposed to be an empty string) for the authorization header, and that I'm not supposed to receive nor use an access token/access secret.
The only things I do have are a consumer key (they call it app token sometimes) and a consumer secret.
Here is how I build my Authorization header :
private static String buildOAuthAuthorization(String method, String request)
throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
String mkmAppToken = APICredentials.appToken;
String mkmAppSecret = APICredentials.appSecret;
String realm = "https://www.mkmapi.eu/ws/v1.1/games";
String oauthVersion = "1.0";
String oauthConsumerKey = mkmAppToken;
String oauthToken = "";
String oauthSignatureMethod = "HMAC-SHA1";
String oauthTimestamp = Long.toString(System.currentTimeMillis() / 1000);
String oauthNonce = Long.toString(System.currentTimeMillis());
String paramString = "oauth_consumer_key=" + oauthConsumerKey
+ "oauth_nonce=" + oauthNonce
+ "oauth_signature_method=" + oauthSignatureMethod
+ "oauth_timestamp=" + oauthTimestamp
+ "oauth_token=" + oauthToken
+ "oauth_version=" + oauthVersion;
String baseString = method + "&" + rawUrlEncode(realm) + "&" + rawUrlEncode(paramString);
String signingKey = rawUrlEncode(mkmAppSecret) + "&";
Mac mac = Mac.getInstance("HMAC-SHA1");
SecretKeySpec secret = new SecretKeySpec(signingKey.getBytes(), mac.getAlgorithm());
mac.init(secret);
byte[] digest = mac.doFinal(baseString.getBytes());
byte[] oauthSignature = Base64.encode(digest, Base64.URL_SAFE);
String authorizationProperty = "OAuth "
+ "realm=\"" + realm + "\", "
+ "oauth_version=\"" + oauthVersion + "\", "
+ "oauth_timestamp=\"" + oauthTimestamp + "\", "
+ "oauth_nonce=\"" + oauthNonce + "\", "
+ "oauth_consumer_key=\"" + oauthConsumerKey + "\", "
+ "oauth_token=\""+ oauthToken + "\", "
+ "oauth_signature_method=\"" + oauthSignatureMethod + "\", "
+ "oauth_signature=\"" + oauthSignature + "\"";
System.out.println(authorizationProperty);
return authorizationProperty;
}
The actual request is in an AsyncTask :
public static class oAuthRequest extends AsyncTask<String, Integer, StringReader> {
private int lastCode;
#Override
protected StringReader doInBackground(String... requestURLs) {
String method = requestURLs[0];
String url = requestURLs[1];
StringReader result = null;
try {
String authProperty = buildOAuthAuthorization(method, url);
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
connection.addRequestProperty("Authorization:", authProperty);
lastCode = connection.getResponseCode();
System.out.println("RESPONSE CODE 1 " + lastCode);
// Get content
BufferedReader rd = new BufferedReader(new InputStreamReader(lastCode == 200 ? connection.getInputStream() : connection.getErrorStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
rd.close();
result = new StringReader(sb.toString());
} catch (NoSuchAlgorithmException | InvalidKeyException | IOException e) {
e.printStackTrace();
}
return result;
}
}
It seems like no matter what I change, I'm always getting a 401.
Things I've tried :
oauthSignature as a String using Base64.encodeToString()
Nonce generation using SecureRandom
With and without the empty oauthToken
Another timestamp generation method (can't remember what though)
signing key with and without app token (theorically I need only the consumer secret, but you never know)
Using HttpsURLConnection instead of HttpURLConnection (the URI start in https, so I thought, hey. But no)
At least 2-3 other different implementations (one who was basically a copy/paste of the Java example in the documentation of course -- it still kind of is one now)
(Probably a lot of things I can't even remember)
At this point I'm wondering if maybe the issue comes from my keys, as I've tried to use the Postman app to test requests with the same results.

Java SFTP (apache vfs2) - password with #

I'm trying to use the org.apache.commons.vfs2 to download a file via SFTP.
The problem is, the password contains the '#' char, so this causes the URI to be parsed incorrectly:
org.apache.commons.vfs2.FileSystemException: Expecting / to follow the hostname in URI
Does anyone has an idea how to get around this issue? (I can't change the password, obviously). This is the code I'm using:
String sftpUri = "sftp://" + userName + ":" + password + "#"
+ remoteServerAddress + "/" + remoteDirectory + fileName;
String filepath = localDirectory + fileName;
File file = new File(filepath);
FileObject localFile = manager.resolveFile(file.getAbsolutePath());
FileObject remoteFile = manager.resolveFile(sftpUri, opts);
localFile.copyFrom(remoteFile, Selectors.SELECT_SELF);

Use an actual URI constructor instead of hand-rolling your own:
String userInfo = userName + ":" + password;
String path = remoteDirectory + filename; // Need a '/' between them?
URI sftpUri = new URI("sftp", userInfo, remoteServerAddress, -1, path, null, null);
...
FileObject remoteFile = manager.resolveFile(sftpUri.toString(), opts);

You need to encode the your password by UriParser.encode(), you can change your code like below:
you code:
String sftpUri = "sftp://" + userName + ":" + password + "#"
+ remoteServerAddress + "/" + remoteDirectory + fileName;
change to:
String sftpUri = "sftp://" + userName + ":" + **UriParser.encode(password, "#".toCharArray())**+ "#"
+ remoteServerAddress + "/" + remoteDirectory + fileName;
Hope it help, thank you.

Twitter API Failed to validate oauth signature and token java

I'm trying to access resources of a twitter account with http protocol, in an Android app. Twitter uses the open authentication standard OAuth for authentication, consequently I'm following the example in https://dev.twitter.com/docs/auth/oauth#Overview.
My problem is :
I whant to acquiring a request token but when I make the request to the endpoint https://api.twitter.com/oauth/request_token the response is "Failed to validate oauth signature and token".
My code :
FOR SIGNATURE STEP
Timestamp and Nonce
oauthTimestamp = String.valueOf(new Date().getTime());
oauthNonce = Base64.encodeToString(oauthTimestamp.getBytes(),Base64.DEFAULT);
//remove /n at the end of the string
oauthNonce = oauthNonce.substring(0, oauthNonce.length() - 1);
Signature base string
String signatureBaseString =
"POST"
+ "&"
+ URLEncoder.encode("https://api.twitter.com/oauth/request_token")
+ "&"
+ URLEncoder.encode("oauth_callback=" + redirectUrl)
+ URLEncoder.encode("&" + "oauth_consumer_key=" + consumerKey)
+ URLEncoder.encode("&" + "oauth_nonce=" + oauthNonce)
+ URLEncoder.encode("&" + "oauth_signature_method=" + "HMAC-SHA1")
+ URLEncoder.encode("&" + "oauth_timestamp=" + oauthTimestamp)
+ URLEncoder.encode("&" + "oauth_version=" + "1.0");
signature call
signature = getSignatureToken(applicationSecret, signatureBaseString, "HmacSHA1");
signature method
private String getSignatureToken(String consumerSecret, String baseString, String algotithm) {
byte[] keyBytes = (consumerSecret+"&").getBytes();
SecretKeySpec secretKeySpec = new SecretKeySpec(keyBytes, algotithm);
Mac mac;
String res = null;
try {
mac = Mac.getInstance(algotithm);
mac.init(secretKeySpec);
byte[] rawHmac = mac.doFinal((baseString).getBytes());
res = android.util.Base64.encodeToString(rawHmac, android.util.Base64.DEFAULT);
res = res.substring(0, res.length() - 1);
System.out.println("MAC : " + res);
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (InvalidKeyException e) {
e.printStackTrace();
}
return res;
}
REQUEST TOKEN STEP
HttpResponse response;
HttpPost authorization = new HttpPost("https://api.twitter.com/oauth/request_token");
final String headerValue =
"OAuth " +
"oauth_nonce=\""+oauthNonce+"\", " +
"oauth_callback=\""+redirectUrl+"\", " +
"oauth_signature_method=\"HMAC-SHA1\", " +
"oauth_timestamp=\""+oauthTimestamp+"\", " +
"oauth_consumer_key=\""+consumerKey+"\", " +
"oauth_signature=\""+URLEncoder.encode(signature)+"\", " +
"oauth_version=\"1.0\"";
authorization.addHeader("Authorization", headerValue);
response = this.httpClient.execute(authorization);
System.out.println(EntityUtils.toString(response.getEntity()));
response = this.httpClient.execute(authorization) give me HTTP/1.1 401 Unauthorized
Any ideas?
Thanks

Well, you'll have to debug this in stages, but one thing I find strange: why are you removing the last character of the HMAC (signature) in getSignatureToken()? That could break the Base64 encoding.

How do I send an image file in a Http POST request? (java)

So I'm writing a small app to dump a directory of images into the user's tumblr blog, using their provided API: http://www.tumblr.com/docs/en/api
I've gotten plaintext posting to work, but now I need to find out how to send an image file in the POST instead of UTF-8 encoded text, and I'm lost. My code at the moment is returning a 403 forbidden error, as if the username and password were incorrect (they're not), and everything else I try gives me a bad request error. I'd rather not have to use external libraries for this if I can. This is my ImagePost class:
public class ImagePost {
String data = null;
String enc = "UTF-8";
String type;
File img;
public ImagePost(String imgPath, String caption, String tags) throws IOException {
//Construct data
type = "photo";
img = new File(imgPath);
data = URLEncoder.encode("email", enc) + "=" + URLEncoder.encode(Main.getEmail(), enc);
data += "&" + URLEncoder.encode("password", enc) + "=" + URLEncoder.encode(Main.getPassword(), enc);
data += "&" + URLEncoder.encode("type", enc) + "=" + URLEncoder.encode(type, enc);
data += "&" + URLEncoder.encode("data", enc) + "=" + img;
data += "&" + URLEncoder.encode("caption", enc) + "=" + URLEncoder.encode(caption, enc);
data += "&" + URLEncoder.encode("generator", "UTF-8") + "=" + URLEncoder.encode(Main.getVersion(), "UTF-8");
data += "&" + URLEncoder.encode("tags", "UTF-8") + "=" + URLEncoder.encode(tags, "UTF-8");
}
public void send() throws IOException {
// Set up connection
URL tumblrWrite = new URL("http://www.tumblr.com/api/write");
HttpURLConnection http = (HttpURLConnection) tumblrWrite.openConnection();
http.setDoOutput(true);
http.setRequestMethod("POST");
http.setRequestProperty("Content-Type", "image/png");
DataOutputStream dout = new DataOutputStream(http.getOutputStream());
//OutputStreamWriter out = new OutputStreamWriter(http.getOutputStream());
// Send data
http.connect();
dout.writeBytes(data);
//out.write(data);
dout.flush();
System.out.println(http.getResponseCode());
System.out.println(http.getResponseMessage());
dout.close();
}
}

I suggest you use MultipartRequestEntity (successor of deprecated MultipartPostMethod) of the Apache httpclient package. With MultipartRequestEntity you can send a multipart POST request including a file. An example is below:
public static void postData(String urlString, String filePath) {
log.info("postData");
try {
File f = new File(filePath);
PostMethod postMessage = new PostMethod(urlString);
Part[] parts = {
new StringPart("param_name", "value"),
new FilePart(f.getName(), f)
};
postMessage.setRequestEntity(new MultipartRequestEntity(parts, postMessage.getParams()));
HttpClient client = new HttpClient();
int status = client.executeMethod(postMessage);
} catch (HttpException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}