We Keep Coding

I am not able to write (put) to Hadoop MiniCluster i created

null: [HTTP/1.1 405 HTTP method
PUT is not supported by this
URL]
Server:
[Jetty(6.1.26.cloudera.4)]
Pragma: [no-cache]
Content-Length: [0]
X-FRAME-OPTIONS: [SAMEORIGIN]
Date: [Wed, 13 Oct 2021
17:05:16 GMT]
'''
private static void putTest() throws IOException {
System.out.println("putTest : Entry");
URL url = new URL("http://localhost:50070/webapps/v1/longdata.txt&op=CREATE");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("PUT");
con.setDoOutput(true);
con.getOutputStream().write("Hey Guys".getBytes(StandardCharsets.UTF_8));
Map<String, List<String>> headers = con.getHeaderFields();
for (String header : headers.keySet())
{
System.out.println(header + ": " + headers.get(header));
}
System.out.println("putTest : Exit");
}
'''

You need two requests - one to the namenode port, which creates the file on the namenode, then that returns a datanode address where you send the second address to write the content stream.
However, you are using Java, so just use hadoop-client dependency with the FileSystem.create method, do not use HttpURLConnection

Getting body bytes from HttpResponse in Netty client

I am trying to create HTTP client using netty and everything work, but i have hard time parsing the body. My pipeline looks like this:
pipeline.addLast(new HttpClientCodec())
pipeline.addLast(new HttpContentDecompressor())
pipeline.addLast(new HttpObjectAggregator(1024*10))
pipeline.addLast(new HttpClientHandler[A](key, metrics))
and client handler (written in scala)
class HttpClientHandler[A: BodyParser](key: AttributeKey[Callback[A]], metrics: Metrics)
extends SimpleChannelInboundHandler[FullHttpResponse]
with LazyLogging {
override def channelRead0(ctx: ChannelHandlerContext, msg: FullHttpResponse): Unit = {
val callback = ctx.channel().attr(key).get()
if (callback != null) {
val response = buildResponse(msg)
callback(response)
} else {
throw new Exception("Callback not present in channel context ... this is a bug")
}
}
private def buildResponse(msg: FullHttpResponse): Either[Throwable, Response[A]] = {
val result = {
try {
val parsedBody = BodyParser[A].parse(msg.content().asReadOnly())
if (msg.status() == HttpResponseStatus.OK) {
Right(Response.Ok(parsedBody))
} else {
Right(Response.Other(msg.status().code(), parsedBody))
}
} catch {
case e: Throwable =>
Left(e)
}
}
result.fold(metrics.bodyParseFailure, metrics.successfulResponse)
result
}
override def exceptionCaught(ctx: ChannelHandlerContext, cause: Throwable): Unit = {
logger.warn(s"error observed for channel ${ctx.channel()}, closing", cause)
ctx.channel().attr(key).get().apply(Left(cause))
}
}
The main issue is that msg.content() also contains Http data (method, version, headers ...) but im only interested in body. What am i doing wrong? Thanks!

That's super strange... msg.content() is a ByteBuf which only should have the payload of the request / response included.

Turns out this was caused by testing this against echo server which repeats the whole http request back, as proven by:
$ curl -X POST localhost:3000/a/b/c -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 3000 (#0)
> POST /a/b/c HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 19 Dec 2020 09:50:56 GMT
< X-Http-Echo-Server-Id: 9fcf5d57-2ac7-4a92-8aed-775879bf3ac2
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Server: Jetty(9.4.24.v20191120)
<
POST /a/b/c HTTP/1.1
Accept: */*
User-Agent: curl/7.64.1
Host: localhost:3000
* Connection #0 to host localhost left intact
* Closing connection 0

Send https get request

I am working on a crawler and I have the following question: it works with simple HTTP requests, but not HTTPS, and I need to make an HTTPS request. I changed the port to 443 and try to send the same request, but I get 400 error. Obviously, I need to change something else, but I do not know what. I open socket and this is how I make the request:
String request
= "GET " + file
+ (file.endsWith("robots.txt") ? " HTTP/1.0\r\n" : " HTTP/1.1\r\n")
// " HTTP/1.1\r\n"
+ "User-Agent: " + CrawlerConfig.USER_AGENT + "\r\n"
// + ((!CrawlerConfig.SAVE_IMAGES) ? "Accept: text/html\r\n" : "")
// + "Accept: text/*\r\n"
+ (file.endsWith("robots.txt") ? "Connection: close\r\n" : "")
+ "Host: " + host + "\r\n" + "\r\n"/*
* + body
*/;
outStream.write(request.getBytes("US-ASCII"));
outStream.flush();

Try to send an OPTIONS request on the resource, needed request headers should be returned, some may be missing.

Why a string in the url is getting changed by few characters after sending the GET request?

Background:
I am trying to use one-api, to send SMS to phones and receive the delivery status. I make a POST request to their servers. Then I extract a string from the JSON response I receive. I use that string to make another GET request to a URL containing that ID. The problem is that the string gets changed when I make the GET request. Although it is same when I extract it from the response, but have no idea why its changing during the course of GET request.
Methodology followed and problem explanation:
The response from initial POST request` :
{"resourceReference":
{"resourceURL":"https:\/\/oneapi-gw.gsma.com:443\/SendSmsService\/OneAPI_REST_v2_0\/routing\/2_0\/smsmessaging\/outbound\/tel:7511\/requests\/998371119"}
}
I extract the ID that I receive in the url(998371119). I use .split to extract as follows:
String tmp = (String)resourceReference.get("resourceURL");
String [] tmp2 = tmp.split("/");
String id = tmp2[(tmp2.length)-1].toString();
System.out.println(id);
// the output is:998371119.
Using this ID I create another url to send a GET request.
String url2 = "https://oneapi-gw.gsma.com/smssend/2_0/smsmessaging/outbound/tel:7511/requests/"+id+"/deliveryInfos"
When I send the GET request, the last 2 digits of ID seems to change automatically and I receive the Response Code as 400 after making a GET request. Following is the error stream:
{ "requestError" :
{ "serviceException" : {
"text" : "Invalid input value for message part requestIdentifier",
"variables" : [
"requestIdentifier", "998371122"
]
}
}
}
Notice how the server interpreted the ID as 998371122 instead of 998371119(my initial requested one). That last 2 digit changed. I've thought a lot but I've no idea why it is happening. Although when I am using curl to send the GET request, everything works fine. So it isn't a server issue. Their is some problem in either How I form the url or extract the ID or make a GET request. Any suggestions? Thanks in advance.
Other Relevant code you might need
This is how I'm making a GET request:
private String getResponseFromGETRequest(String accept, String url) {
URL obj;
StringBuffer response = new StringBuffer();;
HttpURLConnection con;
String authHeaderValue = new String(Base64.encode(credentials.getBytes()));
//Credentals variable is a string storing "username:password"
try {
obj = new URL(url);
con = (HttpURLConnection) obj.openConnection();
con.setRequestProperty ("Authorization", "Basic " + authHeaderValue);
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty ("Accept", accept);
//con.setRequestProperty("Content-Type", accept);
int responseCode = con.getResponseCode();
InputStream ipStream;
if (con.getResponseCode() >= 400) {
ipStream = con.getErrorStream();
} else {
ipStream = con.getInputStream();
}
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(ipStream));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "";
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "";
}
return response.toString();
}
Logcat:
Oct 04, 2013 3:59:07 PM sun.net.www.protocol.http.HttpURLConnection plainConnect
FINEST: ProxySelector Request for https://oneapi-gw.gsma.com/smssend/2_0/smsmessaging/outbound/tel%3A7511/requests
Oct 04, 2013 3:59:07 PM sun.net.www.protocol.http.HttpURLConnection plainConnect
FINEST: Proxy used: DIRECT
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
FINE: sun.net.www.MessageHeader#7f47e35410 pairs: {POST /smssend/2_0/smsmessaging/outbound/tel%3A7511/requests HTTP/1.1: null}{Authorization: Basic ABCDEFGHMYAUTHORIZATIONKEY=}{Accept: application/json}{Content-Type: application/json}{Cache-Control: no-cache}{Pragma: no-cache}{User-Agent: Java/1.7.0_21}{Host: oneapi-gw.gsma.com}{Connection: keep-alive}{Content-Length: 212}
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
FINE: sun.net.www.MessageHeader#75240d4a17 pairs: {null: HTTP/1.1 201 Created}{Date: Fri, 04 Oct 2013 19:59:08 GMT}{Server: Jetty(6.1.x)}{Content-Type: application/json}{Location: https://oneapi-gw.gsma.com:443/SendSmsService/OneAPI_REST_v2_0/routing/2_0/smsmessaging/outbound/tel:7511/requests/998380556}{Host: oneapi-gw.gsma.com}{X-Forwarded-Server: oneapi-gw.gsma.com}{Authorization: Basic ABCDEFGHMYAUTHORIZATIONKEY=}{User-Agent: Java/1.7.0_21}{Accept: application/json}{X-Forwarded-For: 10.90.24.132}{X-Forwarded-Host: oneapi-gw.gsma.com}{breadcrumbId: ID-dtx-prod-apihr01-39903-1371168975552-0-440221}{Vary: Accept-Encoding,User-Agent}{Keep-Alive: timeout=5, max=100}{Connection: Keep-Alive}{Transfer-Encoding: chunked}
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection plainConnect
FINEST: ProxySelector Request for https://oneapi-gw.gsma.com/smssend/2_0/smsmessaging/outbound/tel:7511/requests/998380556/deliveryInfos
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection plainConnect
FINEST: Proxy used: DIRECT
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection writeRequests
FINE: sun.net.www.MessageHeader#71d198cb9 pairs: {GET /smssend/2_0/smsmessaging/outbound/tel:7511/requests/998380556/deliveryInfos HTTP/1.1: null}{Authorization: Basic ABCDEFGHMYAUTHORIZATIONKEY=}{Accept: application/json}{Content-Type: application/json}{Cache-Control: no-cache}{Pragma: no-cache}{User-Agent: Java/1.7.0_21}{Host: oneapi-gw.gsma.com}{Connection: keep-alive}
Oct 04, 2013 3:59:08 PM sun.net.www.protocol.http.HttpURLConnection getInputStream
FINE: sun.net.www.MessageHeader#778671cd15 pairs: {null: HTTP/1.1 400 Bad Request}{Date: Fri, 04 Oct 2013 19:59:09 GMT}{Server: Jetty(6.1.x)}{Content-Type: application/json}{Accept: application/json}{Host: oneapi-gw.gsma.com}{breadcrumbId: ID-dtx-prod-apihr02-48223-1371168511818-0-440001}{X-Forwarded-Host: oneapi-gw.gsma.com}{X-Forwarded-For: 10.90.24.132}{User-Agent: Java/1.7.0_21}{X-Forwarded-Server: oneapi-gw.gsma.com}{Authorization: Basic ABCDEFGHMYAUTHORIZATIONKEY=}{Vary: Accept-Encoding,User-Agent}{Connection: close}{Transfer-Encoding: chunked}
Header: Date : Fri, 04 Oct 2013 19:59:09 GMT
Header: Server : Jetty(6.1.x)
Header: Content-Type : application/json
Header: Accept : application/json
Header: Host : oneapi-gw.gsma.com
Header: breadcrumbId : ID-dtx-prod-apihr02-48223-1371168511818-0-440001
Header: X-Forwarded-Host : oneapi-gw.gsma.com
Header: X-Forwarded-For : 10.90.24.132
Header: User-Agent : Java/1.7.0_21
Header: X-Forwarded-Server : oneapi-gw.gsma.com
Header: Authorization : Basic ABCDEFGHMYAUTHORIZATIONKEY=
Header: Vary : Accept-Encoding,User-Agent
Header: Connection : close
Header: Transfer-Encoding : chunked
Response Code : 400
{
"requestError" : {
"serviceException" : {
"messageId" : "SVC0002",
"text" : "Invalid input value for message part requestIdentifier",
"variables" : [ "requestIdentifier", "998380559" ]
}
}
}
Edit:

Finally after two days it was a strange solution. It turns out parsing was fine and so was the formation of GET and POST request. I just kept 3 seconds delay after I received the response from POST request and before I sent another GET request. This solved it.
try {
Thread.sleep(3000);
} catch(InterruptedException ex) {
Thread.currentThread().interrupt();
}

Bounced mails get Status header

I am trying to read the Status header of a bounced email. This site explains better what I am trying...
The original email is composed by several MultiParts objects, so I am reading it in java code:
private void test(MimeMessage message) throws IOException, MessagingException {
if (message.getContent() != null && message.getContent() instanceof Multipart) {
Multipart content = (Multipart) message.getContent();
for (int i = 0; i < content.getCount(); i++) {
BodyPart bodyPart = content.getBodyPart(i);
Enumeration headers = bodyPart.getAllHeaders();
while(headers.hasMoreElements()){
Header header = (Header) headers.nextElement();
LOGGER.info("Header: " + header.getName() + " value: " + header.getValue());
}
}
}
}
The email part I am analyzing:
Content-Description: Delivery report Content-Type: text/plain;
charset=utf-8 Content-Transfer-Encoding: 7bit
Reporting-MTA: dns; someLink.com
X-Postfix-Queue-ID: EC862F00D0 X-Postfix-Sender: rfc822;
receiver#email.com Arrival-Date: Wed, 7 Aug 2013
13:52:43 +0200 (CEST)
Final-Recipient: rfc822; noexisting#email.com
Original-Recipient: rfc822;noexisting#email.com Action:
failed Status: 5.1.1 Remote-MTA: dns; [somelink.com
Diagnostic-Code: smtp; 550-5.1.1 The email account that you tried to
reach does
not exist. Please try 550-5.1.1 double-checking the recipient's email
address for typos or 550-5.1.1 unnecessary spaces.
In my log file I can see only the 3 first headers:
> Header: Content-Description value: Delivery report
> Header: Content-Type value: text/plain; charset=us-ascii INFO
> Header: Content-Transfer-Encoding value: 7bit
Does anyone know why? How could I get the status header? Thanks

I couldn´t find the Status information in the header, and I will take it from the content. It is not an elegant solution, but at least it works.
If someone finds a better one, please let me know!
Java code:
StringWriter writer = new StringWriter();
IOUtils.copy(bodyPart.getInputStream(), writer);
LOGGER.info("Content inputstream: " + writer.toString());
Logs:
Content inputstream: Reporting-MTA: dns; srvvie-mx3.styria-multi-media.com
X-Postfix-Queue-ID: 2A1A8F00CF X-Postfix-Sender: rfc822;
Arrival-Date: Fri, 9 Aug 2013
11:14:02 +0200 (CEST)
Final-Recipient: rfc822; MAILER-DAEMON#domain.com
Original-Recipient: rfc822;MAILER-DAEMON#domain.com
Action: failed Status: 5.1.1 Remote-MTA: dns;
Diagnostic-Code: smtp; 550 5.1.1 Mailbox
does not exist