We Keep Coding

I am not able to write (put) to Hadoop MiniCluster i created

null: [HTTP/1.1 405 HTTP method
PUT is not supported by this
URL]
Server:
[Jetty(6.1.26.cloudera.4)]
Pragma: [no-cache]
Content-Length: [0]
X-FRAME-OPTIONS: [SAMEORIGIN]
Date: [Wed, 13 Oct 2021
17:05:16 GMT]
'''
private static void putTest() throws IOException {
System.out.println("putTest : Entry");
URL url = new URL("http://localhost:50070/webapps/v1/longdata.txt&op=CREATE");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("PUT");
con.setDoOutput(true);
con.getOutputStream().write("Hey Guys".getBytes(StandardCharsets.UTF_8));
Map<String, List<String>> headers = con.getHeaderFields();
for (String header : headers.keySet())
{
System.out.println(header + ": " + headers.get(header));
}
System.out.println("putTest : Exit");
}
'''

You need two requests - one to the namenode port, which creates the file on the namenode, then that returns a datanode address where you send the second address to write the content stream.
However, you are using Java, so just use hadoop-client dependency with the FileSystem.create method, do not use HttpURLConnection

Getting body bytes from HttpResponse in Netty client

I am trying to create HTTP client using netty and everything work, but i have hard time parsing the body. My pipeline looks like this:
pipeline.addLast(new HttpClientCodec())
pipeline.addLast(new HttpContentDecompressor())
pipeline.addLast(new HttpObjectAggregator(1024*10))
pipeline.addLast(new HttpClientHandler[A](key, metrics))
and client handler (written in scala)
class HttpClientHandler[A: BodyParser](key: AttributeKey[Callback[A]], metrics: Metrics)
extends SimpleChannelInboundHandler[FullHttpResponse]
with LazyLogging {
override def channelRead0(ctx: ChannelHandlerContext, msg: FullHttpResponse): Unit = {
val callback = ctx.channel().attr(key).get()
if (callback != null) {
val response = buildResponse(msg)
callback(response)
} else {
throw new Exception("Callback not present in channel context ... this is a bug")
}
}
private def buildResponse(msg: FullHttpResponse): Either[Throwable, Response[A]] = {
val result = {
try {
val parsedBody = BodyParser[A].parse(msg.content().asReadOnly())
if (msg.status() == HttpResponseStatus.OK) {
Right(Response.Ok(parsedBody))
} else {
Right(Response.Other(msg.status().code(), parsedBody))
}
} catch {
case e: Throwable =>
Left(e)
}
}
result.fold(metrics.bodyParseFailure, metrics.successfulResponse)
result
}
override def exceptionCaught(ctx: ChannelHandlerContext, cause: Throwable): Unit = {
logger.warn(s"error observed for channel ${ctx.channel()}, closing", cause)
ctx.channel().attr(key).get().apply(Left(cause))
}
}
The main issue is that msg.content() also contains Http data (method, version, headers ...) but im only interested in body. What am i doing wrong? Thanks!

That's super strange... msg.content() is a ByteBuf which only should have the payload of the request / response included.

Turns out this was caused by testing this against echo server which repeats the whole http request back, as proven by:
$ curl -X POST localhost:3000/a/b/c -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 3000 (#0)
> POST /a/b/c HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 19 Dec 2020 09:50:56 GMT
< X-Http-Echo-Server-Id: 9fcf5d57-2ac7-4a92-8aed-775879bf3ac2
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Server: Jetty(9.4.24.v20191120)
<
POST /a/b/c HTTP/1.1
Accept: */*
User-Agent: curl/7.64.1
Host: localhost:3000
* Connection #0 to host localhost left intact
* Closing connection 0

EWS SOAP request fails on updateItem if Category string has accented chars

I would like to know how can I post the below request via a java/groovy script:
SOAP Request sent (size: 1232)
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<RequestServerVersion xmlns="http://schemas.microsoft.com/exchange/services/2006/types" Version="Exchange2013_SP1"></RequestServerVersion>
</soap:Header>
<soap:Body>
<UpdateItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" MessageDisposition="SaveOnly" ConflictResolution="AutoResolve">
<ItemChanges>
<t:ItemChange xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
<t:ItemId Id="AAMkADVmY2M3YTNmLTRmNTAtNDgxOS05N2ZhLWMyNTc0YWZlMDhlNwBGAAAAAAAQBgKUNgxZT6CYzuo2SsPlBwDW6sia8LrPRrmNln3i877OAAAAAAEMAAAT3G5bCC7PSJ3ZbmUh540OAACBI4lIAAA=" ChangeKey="CQAAABYAAAAT3G5bCC7PSJ3ZbmUh540OAACBK17k"></t:ItemId>
<t:Updates>
<t:SetItemField>
<t:FieldURI FieldURI="item:Categories"></t:FieldURI>
<t:Message>
<t:Categories>
<t:String>ÁÉÍÓÖÚÜ</t:String>
</t:Categories>
</t:Message>
</t:SetItemField>
</t:Updates>
</t:ItemChange>
</ItemChanges>
</UpdateItem>
</soap:Body>
</soap:Envelope>
HTTP/1.1 500 Internal Server Error
Server: Microsoft-IIS/7.5.
request-id: 9baa3ecb-99c1-42a9-98b6-1b5b74af4c9d.
X-AspNet-Version: 4.0.30319.
Cache-Control: private.
Content-Type: text/xml; charset=utf-8.
Date: Fri, 11 Dec 2020 01:49:20 GMT.
Transfer-Encoding: chunked.
Persistent-Auth: true.
Connection: close.
Set-Cookie: exchangecookie=7fe34835e0d84ccaaa4bafdbd70b424a; expires=Sat, 11-Dec-2021 01:49:20 GMT; path=/; HttpOnly.
Set-Cookie: X-BackEndCookie=S-1-5-21-1275210071-2000478354-682003330-2272537=u56Lnp2ejJqBnZmdyZ7MyM/SmcvIyNLLysaa0p6dm5zSyMnIzs7GyJmeyJyagYHNz83O0s/O0s7Pq8/OxcvGxc3P; expires=Sun, 10-Jan-2021 01:49:20 GMT; path=/ews; secure; HttpOnly.
X-Powered-By: ASP.NET.
Same request works fine using SOAPe.
Same request passess thru fine when sent with no special accented chars.
I tried encoding chars to no avail.

Found this which works:
static String escapeSpecialChars(String inputData){
def value = inputData.toString();
Pattern p = Pattern.compile('[^#|()-_&\'"!. A-Za-z0-9]');
Matcher m = p.matcher(value);
boolean b = m.find();
if (b == true){
def input = value;
def output=''
for(int l=0;l<input.length();l++){
char temp = input.getAt(l);
int ascii = temp
boolean check = p.matcher(temp.toString());
if(check == true){
output += '&#'+ascii+';';
println '&#'+ascii+';'
}else{
output += temp;
}
}
value = output;
}
inputData = value;
return inputData; }
but a more elegant solution might be wrapping the categories in a CDATA like so:
cat = cat ==~ /[#|()-_&\'\"!. A-Za-z0-9]*/ ? "$cat" : "<![CDATA[$cat]]>"

how to extract data from restclient.entity.UserEntities in openfire rest API?

I am using this : Openfire Rest API
Now I am fetching users and groups using a java file.
In response I was expecting XML data, but it shows me strange data.
I am new to Java so I don't know how to extract data from this.
My Code is :
package bizrtc;
import api.restclient.RestClient;
import api.restclient.RestApiClient;
import api.restclient.entity.AuthenticationToken;
import api.restclient.entity.UserEntities;
import api.restclient.entity.GroupEntities;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
*
* #author Rajan
*/
public class Bizrtc
{
/**
* #param args the command line arguments
*/
public static void main(String[] args)
{
// TODO code application logic here
AuthenticationToken authenticationToken = new AuthenticationToken("cn1ed9s8yEf5woQV");
// Set Openfire settings (9090 is the port of Openfire Admin Console)
RestApiClient restApiClient = new RestApiClient("192.168.50.50", 9090, authenticationToken);
// restApiClient.getUsers();
UserEntities users = restApiClient.getUsers();
System.out.println("The Groups are as below: "+restApiClient.getGroups());
System.out.println("Now fetching data from openfire Server");
System.out.println("The data is *******************************" + users.toString());
}
}
And when I run the program I get:
Dec 23, 2016 3:58:43 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 1 * Sending client request on thread main
1 > GET http://192.168.50.50:9090/plugins/restapi/v1/groups
1 > Authorization: cn1ed9s8yEf5woQV
1 > Content-Type: application/xml
Dec 23, 2016 3:58:44 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 1 * Client response received on thread main
1 < 200
1 < Access-Control-Allow-Credentials: true
1 < Access-Control-Allow-Headers: origin, content-type, accept, authorization
1 < Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, HEAD
1 < Access-Control-Allow-Origin: *
1 < Content-Length: 3664
1 < Content-Type: application/xml
1 < Date: Fri, 23 Dec 2016 09:53:38 GMT
1 < Expires: Thu, 01 Jan 1970 00:00:00 GMT
1 < Set-Cookie: JSESSIONID=1bt213yrejbmfkpyfs53snplm;Path=/;HttpOnly
1 < X-Frame-Options: deny
The Groups are as below: api.restclient.entity.GroupEntities#1165b38
Now fetching data from openfire Server
The data is *******************************api.restclient.entity.UserEntities#4c12331b
How To print this users in XML or better to ARRAY format ??
How do i get the users from this response:api.restclient.entity.GroupEntities#1165b38
Do i have to convert this to string or something like that ?

Take a look at UserEntities Java code:
#XmlRootElement(
name = "users"
)
public class UserEntities {
List<UserEntity> users;
public UserEntities() {
}
public UserEntities(List<UserEntity> users) {
this.users = users;
}
#XmlElement(
name = "user"
)
public List<UserEntity> getUsers() {
return this.users;
}
public void setUsers(List<UserEntity> users) {
this.users = users;
}
}
It's a single POJO class with a list of users and mapped with JAXB annotations. This means you can easily convert your object to XML, JSON or whatever the lib enables.
XML way:
JAXBContext jaxbContext = JAXBContext.newInstance(UserEntities.class);
Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
jaxbMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
StringWriter sw = new StringWriter();
jaxbMarshaller.marshal(users, sw);
String xmlString = sw.toString();
System.out.println(xmlString);
And if you want an array of UserEntity's, you already have its List:
final UserEntity[] arrayUsers = (UserEntity[]) users.getUsers().toArray();
Return example:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<users>
<user>
<username>d</username>
<name>e</name>
<email>a#a.com</email>
<password>pass</password>
</user>
</users>

Uploading a file via Post Request

I am attempting to upload images to SmugMug via HTTP Post as per their documentation. I have all the headers correct, but the part that is confusing me is setting the binary data in the body as stated:
This method requires a POST request with the binary data in the body
and all other metadata in the headers.
I have tried:
SMResponse response = builder.post(SMResponse.class, Files.readAllBytes(image.toPath()));
SMResponse response = builder.post(SMResponse.class, new String(Files.readAllBytes(image.toPath())));
SMResponse response = builder.post(SMResponse.class, new String(Base64.encode(Files.readAllBytes(image.toPath()))));
SMResponse response = builder.post(SMResponse.class, Base64.encode(Files.readAllBytes(image.toPath())));
My best guess was the first one would work, but all of these return:
{"stat":"fail","method":"smugmug.images.upload","code":5,"message":"system error"}
Here is the full method that does the uploading, in case I missed something:
public boolean upload(File image, int albumId, String caption, String keywords,
Boolean hidden, Integer imageId, Integer altitude, Float latitude,
Float longitude, boolean pretty) throws IOException, InvalidKeyException, NoSuchAlgorithmException, SmugMugException {
logger.debug("upload() called");
byte[] imageBytes = Files.readAllBytes(image.toPath());
WebResource resource = SmugMugAPI.CLIENT.resource("http://upload.smugmug.com/");
LoggingFilter logFilter = new LoggingFilter();
resource.addFilter(logFilter);
OAuthSecrets secrets = new OAuthSecrets().consumerSecret(smugmug.getConsumerSecret());
OAuthParameters oauthParams = new OAuthParameters().consumerKey(smugmug.getCosumerKey()).
signatureMethod("HMAC-SHA1").version("1.0");
// Create the OAuth client filter
OAuthClientFilter filter = new OAuthClientFilter(SmugMugAPI.CLIENT.getProviders(), oauthParams, secrets);
// Add the filter to the resource
if (smugmug.getToken() != null){
secrets.setTokenSecret(smugmug.getToken().getSecret());
oauthParams.token(smugmug.getToken().getId());
}
resource.addFilter(filter);
WebResource.Builder builder = resource.getRequestBuilder();
//User agent
builder = builder.header("User-Agent", smugmug.getAppName());
//API Version header
builder = builder.header("X-Smug-Version", "1.3.0");
//Response Type header
builder = builder.header("X-Smug-ResponseType", "JSON");
//Content-Length header
builder = builder.header("Content-Length", Long.toString(image.length()));
//Content-MD5 header
builder = builder.header("Content-MD5", DigestUtils.md5Hex(imageBytes));
//X-Smug-FileName header
builder = builder.header("X-Smug-FileName", image.getName());
//X-Smug-AlbumID header
builder = builder.header("X-Smug-AlbumID", Integer.toString(albumId));
//X-Smug-Caption header
if(caption != null){
builder = builder.header("X-Smug-Caption", caption);
}
//X-Smug-Caption header
if(keywords != null){
builder = builder.header("X-Smug-Keywords", keywords);
}
//X-Smug-Hidden header
if(hidden != null){
builder = builder.header("X-Smug-Hidden", hidden.toString());
}
//X-Smug-ImageID header
if(imageId != null){
builder = builder.header("X-Smug-ImageID", imageId.toString());
}
//X-Smug-Altitude header
if(altitude != null){
builder = builder.header("X-Smug-Altitude", altitude.toString());
}
//X-Smug-Latitude header
if(latitude != null){
builder = builder.header("X-Smug-Latitude", latitude.toString());
}
//X-Smug-Latitude header
if(longitude != null){
builder = builder.header("X-Smug-Longitude", longitude.toString());
}
//X-Smug-Pretty header
if(pretty){
builder = builder.header("X-Smug-Pretty", Boolean.toString(pretty));
}
SMResponse response = builder.post(SMResponse.class, new String(imageBytes));
if (!"ok".equals(response.getStat())) {
throw new SmugMugException(response);
}
return true;
}
Where have I gone wrong?
Tried just to see the response:
SMResponse response = builder.entity(image).post(SMResponse.class);
It actually sent back a blank response (no json) which is odd in itself, as I would have expected some message back. Here is the output:
Nov 21, 2012 11:55:48 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 1 * Client in-bound response
1 < 200
1 < Edge-Control: no-store
1 < X-SmugMug-Hiring: How to love what you do: http://www.smugmug.com/jobs/
1 < Date: Thu, 22 Nov 2012 05:55:48 GMT
1 < Content-Length: 0
1 < X-SmugMug-Values: 4/4 - It's the product, stupid
1 < Expires: Thu, 22 Nov 2012 05:55:49 GMT
1 < Connection: keep-alive
1 < Content-Type: application/json; charset=utf-8
1 < X-Powered-By: SmugMug/0.9
1 < Server: Apache
1 < Cache-Control: private, no-store, no-cache, max-age=1, must-revalidate
1 <

I am not exactly sure what happened, but I was able to get it working after finding the Upload Log in the SmugMug Account Settings (To get there go to Tools -> Account Settings -> Stats -> Uploads -> Details). Note that in the upload log there is a toggle to show only errors or all uploads.
Now on to the actual answer to how to set the actual "body" of the Post Request. The actual format should have been the first one I posted:
SMResponse response = builder.post(SMResponse.class, Files.readAllBytes(image.toPath()));
So either I messed up, and thought it was not working when it was, there was a problem on smugmug's end at the time, or there was something else in my code that was wrong, that got fixed in the process of me trying to fix this non-issue.