Problem statement
I think the title says it all: I'm looking for the way to parse a String containing the body part of a multipart/form-data HTTP request. I.e. the contents of the string would look something like this:
--xyzseparator-blah
Content-Disposition: form-data; name="param1"
hello, world
--xyzseparator-blah
Content-Disposition: form-data; name="param2"
42
--xyzseparator-blah
Content-Disposition: form-data; name="param3"
blah, blah, blah
--xyzseparator-blah--
What I'm hoping to obtain, is a parameters map, or something similar, like this.
parameters.get("param1"); // returns "hello, world"
parameters.get("param2"); // returns "42"
parameters.get("param3"); // returns "blah, blah, blah"
parameters.keys(); // returns ["param1", "param2", "param3"]
Further criteria
It would be best if I don't have to supply the separator (i.e. xyzseparator-blah in this case), but I can live with it if I do have to.
I'm looking for a library based solution, possibly from a main stream library (like "Apache Commons" or something similar).
I want to avoid rolling my own solution, but at the current stage, I'm afraid I will have to. Reason: while the example above seems trivial to split/parse with some string manipulation, real multipart request bodies can have many more headers. Besides that, I do not want to re-invent (and much less re-test!) the wheel :)
Alternative solution
If there were a solution, which satisfies the above criteria, but whose input is an Apache HttpRequest, instead of a String, that would be acceptable too.
(Basically I do receive an HttpRequest, but the in-house library I'm using is built such, that it extracts the body of this request as a String, and passes that to the class responsible for doing the parsing. However, if need be, I could also work directly on the HttpRequest.)
Related questions
No matter how I try to find an answer through Google, here on SO, and on other forums too, the solution seems to be always to use commons fileupload to go through the parts. E.g.: here, here, here, here, here...
However, parseRequest method, used in that solution, expects a RequestContext, which I do not have (only HttpRequest).
The other way, also mentioned in some of the above answers, is getting the parameters from the HttpServletRequest (but again, I only have HttpRequest).
EDIT: In other words: I could include Commons Fileupload (I have access to it), but that would not help me, because I have an HttpRequest, and the Commons Fileupload needs RequestContext. (Unless there is an easy way to convert from HttpRequest to RequestContext, which I have overlooked.)
You can parse your String using Commons FileUpload by wrapping it in a class implementing 'org.apache.commons.fileupload.UploadContext', like below.
I recommend wrapping the HttpRequest in your proposed alternate solution instead though, for a couple of reasons. First, using a String means that the whole multipart POST body, including the file contents,needs to fit into memory. Wrapping the HttpRequest would allow you to stream it, with only a small buffer in memory at one time. Second, without the HttpRequest, you'll need to sniff out the multipart boundary, which would normally be in the 'Content-type' header (see RFC1867).
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUpload;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
public class MultiPartStringParser implements org.apache.commons.fileupload.UploadContext {
public static void main(String[] args) throws Exception {
String s = new String(Files.readAllBytes(Paths.get(args[0])));
MultiPartStringParser p = new MultiPartStringParser(s);
for (String key : p.parameters.keySet()) {
System.out.println(key + "=" + p.parameters.get(key));
}
}
private String postBody;
private String boundary;
private Map<String, String> parameters = new HashMap<String, String>();
public MultiPartStringParser(String postBody) throws Exception {
this.postBody = postBody;
// Sniff out the multpart boundary.
this.boundary = postBody.substring(2, postBody.indexOf('\n')).trim();
// Parse out the parameters.
final FileItemFactory factory = new DiskFileItemFactory();
FileUpload upload = new FileUpload(factory);
List<FileItem> fileItems = upload.parseRequest(this);
for (FileItem fileItem: fileItems) {
if (fileItem.isFormField()){
parameters.put(fileItem.getFieldName(), fileItem.getString());
} // else it is an uploaded file
}
}
public Map<String,String> getParameters() {
return parameters;
}
// The methods below here are to implement the UploadContext interface.
#Override
public String getCharacterEncoding() {
return "UTF-8"; // You should know the actual encoding.
}
// This is the deprecated method from RequestContext that unnecessarily
// limits the length of the content to ~2GB by returning an int.
#Override
public int getContentLength() {
return -1; // Don't use this
}
#Override
public String getContentType() {
// Use the boundary that was sniffed out above.
return "multipart/form-data, boundary=" + this.boundary;
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(postBody.getBytes());
}
#Override
public long contentLength() {
return postBody.length();
}
}
Related
I'm using Jackson 2.4 in Java to do some JSON legwork. I make a call to a remote server with Apache HttpGet, deserialize the results with Jackson into a POJO, manipulate those results, and then serialize them with Jackson to push back to a remote server with HttpPost.
The issue I'm finding is that Jackson is translating unicode literals into unicode characters, which I need it not to do thanks to encoding issues on each end. For example, I might have this in the JSON:
"field1": "\u00a2"
But Jackson is converting the "\u00a2" to "ยข" when it's deserialized, which causes problems with the remote server. It has to be maintained as escaped unicode. If I use something like Apache EntityUtils (specifying UTF-8) or even make the call from my web browser to get the data, the escaped unicode is preserved, so I know that it's coming in properly from the server. If I have Jackson consume the input stream from the entity on the response, it does the conversion automatically.
I've tried writing with a JsonGenerator that is explicitly set to UTF-8 to write to the HttpPost. It didn't work, remote server still rejected it. I've dug through the configuration options for ObjectMapper and JsonParser, but I don't see anything that would override this behavior. Escaping non-ASCII, sure, but that's not what I need to do here. Maybe I'm missing something obvious, but I can't get Jackson to deserialize this string without replacing the escaped unicode.
EDIT: Well, my bad, the only literals having problems have 3 or 5 leading slashes, not just one. That's some screwiness, but Java seems to be what's unpacking it by default during the deserialization, even if the raw text that came back from the server preserves it. Still not sure how to get Java to preserve this without checking an insane amount of text.
What you are expecting is outside scope of Jackosn. It's java that converts the String while reading it. For same reason, if you have a properties file with value \u00a2 and read it using jdk API, you will get converted value. Depending on the file size, either you can double escape char \ before passing the string to Json or "escape" the string back using your Deserializer (only for string) and something like below:
Thank you
package com.test.json;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.module.SimpleModule;
import java.io.IOException;
import java.util.Map;
public class Jackson {
static ObjectMapper _MAPPER = new ObjectMapper();
public static void main(String[] args) throws Exception {
String json = "{\"field1\": \"\\u00a2\",\"field2\": \"\\u00a2 this\",\"numberField\": 121212}";
SimpleModule testModule
= new SimpleModule("StOvFl", _MAPPER.version()).addDeserializer(String.class,
new UnEscapedSerializaer());
_MAPPER.registerModule(testModule);
Map m = _MAPPER.readValue(json, new TypeReference<Map<String, Object>>() {
});
System.out.println("m" + m);
}
}
class UnEscapedSerializaer extends JsonDeserializer<String> {
#Override
public String deserialize(JsonParser jp, DeserializationContext ctxt)
throws IOException, JsonProcessingException {
String s = jp.getValueAsString();
return org.apache.commons.lang.StringEscapeUtils.StringEscapeUtils.escapeJava(s);
}
}
Another way to custom Jackson's behavior is customized JsonParser. See jackson's source code of JsonFactory, ReaderBasedJsonParser;
The key methond is _finishString2() which is used to do 'decodeEscaped', so we can write a JsonParser extends ReaderBasedJsonParser and override the _finishString2 method:
public class MyJsonParser extends ReaderBasedJsonParser {
#Override
protected void _finishString2() throws IOException {
char[] outBuf = _textBuffer.getCurrentSegment();
int outPtr = _textBuffer.getCurrentSegmentSize();
final int[] codes = _icLatin1;
final int maxCode = codes.length;
while (true) {
if (_inputPtr >= _inputEnd) {
if (!loadMore()) {
_reportInvalidEOF(": was expecting closing quote for a string value");
}
}
char c = _inputBuffer[_inputPtr++];
int i = (int) c;
if (i < maxCode && codes[i] != 0) {
if (i == INT_QUOTE) {
break;
} else {
//c = _decodeEscaped();
//do nth
}
}
// Need more room?
if (outPtr >= outBuf.length) {
outBuf = _textBuffer.finishCurrentSegment();
outPtr = 0;
}
// Ok, let's add char to output:
outBuf[outPtr++] = c;
}
_textBuffer.setCurrentLength(outPtr);
}
public static void main(String[] args) throws IOException {
String json = "{\"field1\": \"\\u00a2\",\"field2\": \"\\u00a2 this\",\"numberField\": 121212}";
ObjectMapper objectMapper = new ObjectMapper(new MyJsonParserFactory());
Object o = objectMapper.readValue(json, Object.class);
System.out.println(o);
}
}
Full demo code here
Given I have a class that uses some kind of searcher to get and display a list of URLs, like this:
package com.acme.displayer;
import com.acme.searcher.SearcherInterface;
class AcmeDisplayer {
private SearcherInterface searcher;
public AcmeDisplayer(SearcherInterface searcher) {
this.searcher = searcher;
}
public void display() {
List<String> urls = searcher.getUrls();
for (String url : urls) {
System.out.println(url);
}
}
}
Whereas the SearcherInterface looks like the following:
package com.acme.searcher;
public interface SearcherInterface {
List<String> getUrls();
}
There's multiple implementations of these searchers. (One, for instance, only returns a hardcoded list of Strings for testing purposes).
Another one, however, performs HTTP Requests to whatever API and parses the response for URLs, like so:
package com.acme.searcher.http;
import com.acme.searcher.SearcherInterface;
public class HttpSearcher implements SearcherInterface {
private RequestPerformerInterface requestPerformer;
private ParserInterface parser;
public HttpSearcher(RequestPerformerInterface requestPerformer, ParserInterface parser) {
this.requestPerformer = requestPerformer;
this.parser = parser;
}
List<String> getUrls() {
InputStream stream = requestPerformer.performRequest();
return parser.parse(stream);
}
}
The splitting of such an HTTP request is done because of seperation of concerns.
However, this is leading to a problem: A Parser might only be built for a certain API, which is represented by a certain RequestPerformer. So they need to be compatible. I've fiddled around with generic types for such a structure now, i.e. having a TypeInterface that both arguments of HttpSearchers constructor should implement, but I didn't get it working... Another approach would be to just implement a check in one class if the other one is compatible with it, but that seems ugly.
Is there any way to achieve such a grouping of RequestPerformers and Parsers by the API they're handling? Or is there something wrong with the architecture itself?
Your HttpSearcher seems like such a device to group these 2 together. You could create a factory class that returns HttpSearcher and other classes like it, and code that factory to group the compatible RequestPerformers and Parsers together.
The reason why I wouldn't advice leveraging the type system, e.g. through generics, is that the type InputStream can guarantee nothing about the format/type of data it holds. Separating the responsibility of getting the raw data, and parsing seems like a good idea, but you will still have to 'manually' group the compatible types together, because only you know what format/type of data the InputStream will hold.
I'm developing desktop software with JavaFX and Java Spark which is basically a barebones framework for developing web apps, but I'm trying to use it strictly to put/get sensitive methods and variables on a server so that users can't access them. REST seems to be the correct approach but I'm struggling to understand 2 interrelated REST concepts. The Spark docs are light but I've integrated the few good tutorials into a working demo below but I've hit a wall. I'll briefly explain:
With the help of Postman I've been able to put a few records onto the server by using a path of http://localhost:4567/secrets and Body of:
{
"title" : "demofield1",
"content" : "12345"
}
Each record contains a title as an identifier (demofield1) and content as the sensitive data that should remain hidden from users at all times (12345). It's pretty trivial to put these strings onto a server and then get them by using title as a parameter, shown below. The demo code simply has a Model class for creating and returning a record (secret), a JSON conversion method, and a get and put Spark method. Secrets are stored locally in a HashMap for now, I'm assuming a real app would simply swap in a server DB.
The get method works as expected, returning the correct JSON record and storing the content as a String with this line: String secretString = model.getCertainSecret(title).getContent();
With that said...
Questions (partial answers fully appreciated too):
secretString above now holds a confidential value (12345) which is obtained using a supposedly secure REST method. But couldn't a user simply reverse-engineer my source code and write System.out.println(secretString) and have that 12345 revealed? I don't understand how a simple string is protected after retrieving it from the server, despite not being explicitly shown. The code seems correct yet the value is easily obtainable. What am I missing?
How do you put entire java methods on a server? A lot of code I need to protect isn't just strings but methods containing Tasks, Platform.runLater()->, and needs to interact with other desktop software. For example, one of my methods uses JACOB to identify when a certain area of a third-party software is clicked. I can't even fathom what a get/put would look like in that context.
My assumption was that a server-side DB would store all content from my put requests, but I don't understand how it stores and returns a method? Should I be reading about servlets or SaaS or something? I'm focused on desktop users.
Code:
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import lombok.Data;
import org.apache.log4j.BasicConfigurator;
import java.io.IOException;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;
import static spark.Spark.get;
import static spark.Spark.put;
public class ServerDemo
{
private static final int HTTP_BAD_REQUEST = 400;
#Data
static class NewSecretPayload {
private String title;
private String content;
public boolean isValid() {
return title != null && !title.isEmpty();
}
}
public static class Model {
private int nextId = 1;
private Map<String, Secret> secrets = new HashMap<>();
#Data
class Secret {
private int id;
private String title;
private String content;
}
public int createSecret(String title, String content){
int id = nextId++;
Secret secret = new Secret();
secret.setId(id);
secret.setTitle(title);
secret.setContent(content);
secrets.put(title, secret);
return id;
}
public Secret getCertainSecret(String titleToUse){
if(null != secrets.get(titleToUse)){
return secrets.get(titleToUse);
}else{
return null;
}
}
}
public static String dataToJson(Object data) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
StringWriter sw = new StringWriter();
mapper.writeValue(sw, data);
return sw.toString();
} catch (IOException e) {
throw new RuntimeException("IOException from a StringWriter?");
}
}
public static void main(String[] args) {
Model model = new Model();
BasicConfigurator.configure();
put("/secrets", (request, response) -> {
try {
ObjectMapper mapper = new ObjectMapper();
NewSecretPayload creation = mapper.readValue(request.body(), NewSecretPayload.class);
if (!creation.isValid()) {
response.status(HTTP_BAD_REQUEST);
return "";
}
int id = model.createSecret(creation.getTitle(), creation.getContent());
response.status(200);
response.type("application/json");
return id;
} catch (JsonParseException jpe) {
response.status(HTTP_BAD_REQUEST);
return "";
}
});
get("/secrets/:title", (req, res) -> {
String title = req.params(":title");
if (model.getCertainSecret(title) != null) {
res.status(200);
res.type("application/json");
String secretString = model.getCertainSecret(title).getContent();
return dataToJson(model.getCertainSecret(title));
}
res.status(400);
return new ResponseError("No user with title "+title+" was found", title);
});
}
}
Lets dig down to your first problem "Keeping string secret" :--
Restrict : The simplest way is not to provide the data to malicious user.
Masking : Mask the data you are providing to end user. You will have the original data mapped to masked data. You will provide masked data to end user. Here the end user can never get the original data as it is a one way process. When end user sends masked-data you can always retrieve the original data from it.
Encrypting : If the end user needs to see the data you can encrypt it and send it. You can make a sanity check of your code before decrypting the data. The sanity check can give you idea if the code is ever modified. If code fails the sanity check you can always exit the process.
I am trying to convert XML String in the below program to JSON String.
I am able to convert it from file but not from the string.
Any idea about this?
package com.tda.topology;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import org.apache.camel.Exchange;
import org.apache.camel.dataformat.xmljson.XmlJsonDataFormat;
public class Demo2 {
public static void main(String[] args) throws Exception {
String xmlstring = "<soapenv:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ser=\"http://services.web.post.list.com\"><soapenv:Header><authInfo xsi:type=\"soap:authentication\" xmlns:soap=\"http://list.com/services/SoapRequestProcessor\"><!--You may enter the following 2 items in any order--><username xsi:type=\"xsd:string\">user#email.com</username><password xsi:type=\"xsd:string\">password</password></authInfo></soapenv:Header></soapenv:Envelope>";
XmlJsonDataFormat xmlJsonDataFormat = new XmlJsonDataFormat();
xmlJsonDataFormat.setEncoding("UTF-8");
xmlJsonDataFormat.setForceTopLevelObject(true);
xmlJsonDataFormat.setTrimSpaces(true);
xmlJsonDataFormat.setRootName("newRoot");
xmlJsonDataFormat.setSkipNamespaces(true);
xmlJsonDataFormat.setRemoveNamespacePrefixes(true);
Exchange exchange;
//exchange.setIn(in);
InputStream stream = new ByteArrayInputStream(xmlstring.getBytes(StandardCharsets.UTF_8));
//xmlJsonDataFormat.getSerializer().readFromStream(stream).toString();
//xmlJsonDataFormat.marshal(exchange, graph, stream);
}
}
You need to call start on your xmlJsonDataFormat object and add xom jar to your class path (if it's not already there). This is what worked for me:
xmlJsonDataFormat.start();
String json = xmlJsonDataFormat.getSerializer().readFromStream(stream).toString();
I was able to work this out through looking in the source. getSerialiser was returning null and so I searched in xmlJsonDataFormat for where the serializer was initialised and it's done so by the doStart method which is called in the super class's start method.
Disclaimer: not sure you're supposed to use XmlJsonDataFormat like this, it's usually meant for use in a camel route: from("direct:marshal").marshal(xmlJsonFormat).to("mock:json"); but I don't know your specific use case.
I am trying to learn the Play Framework 2.4. I am trying to get the time it takes to access different webpages asynchronously using Promise. Below is the code for that:
final long start = System.currentTimeMillis();
F.Function<WSResponse,Long> timing = new F.Function<WSResponse, Long>() {
#Override
public Long apply(WSResponse wsResponse) throws Throwable {
return System.currentTimeMillis()-start;
}
};
F.Promise<Long> google = ws.url("http://www.google.com").get().map(timing);
F.Promise<Long> yahoo = ws.url("http://www.yahoo.com").get().map(timing);
F.Promise<Long> bing = ws.url("http://www.bing.com").get().map(timing);
As you can see I am using the get function to get the requested pages and putting the result in a Future Promise. Then I convert/map it to long. What I am not able to do is how do I compose these three promises into one and once all of the three promises are redeemed convert/map it to json and return the result. In earlier versions of Play it could have been done by F.Promise.waitAll(google,yahoo,bing).map(...) however I am unable to do it in Play 2.4. Please advice
EDIT1: Based on the answer below i used sequence like below:
return F.Promise.sequence(google, yahoo, bing).map(new F.Function<List<Long>, Result>() {
#Override
public Result apply(List<Long> longs) throws Throwable {
Map<String, Long> data = new HashMap<String, Long>();
data.put("google", google.get());
data.put("yahoo", yahoo.get());
data.put("bing", bing.get());
return ok(Json.toJson(data));
}
});
However, i am getting error that google.get() method cannot be resolved and that Json cannot be applied. What am i missing here?
EDIT 2. The Json error was fixed by using return ok((JsonNode) Json.toJson((Writes<Object>) data)); However, i am still not able to resolve the earlier error that google.get() method cannot be resolved in the line data.put("google", google.get());
EDIT 3. It seems Play2.4 has no get() method which returns the value of a Promise once it has been redeemed. What should i use then?
waitAll has been replaced with F.Promise.sequence.
From the docs
public static <A> F.Promise<java.util.List<A>> sequence(java.lang.Iterable<F.Promise<A>> promises)
Combine the given promises into a single promise for the list of results. The sequencing operations are performed in the default ExecutionContext.
Parameters:
promises - The promises to combine
Returns:
A single promise whose methods act on the list of redeemed promises
Update
Regarding the second half of the question, you don't need to call .get() because the promises have already completed.
In fact, you can get rid of the individual promise variables and just pass them directly into the sequence. The resulting list will contain results in the same order (Google first, then Yahoo, then Bing, in this case).
The whole thing should look something like this:
package controllers;
import java.util.HashMap;
import java.util.Map;
import play.libs.F;
import play.libs.Json;
import play.libs.ws.WS;
import play.libs.ws.WSResponse;
import play.mvc.Controller;
import play.mvc.Result;
import play.mvc.Results;
public class Application extends Controller {
public F.Promise<Result> index() {
final long start = System.currentTimeMillis();
final F.Function<WSResponse,Long> timing = response -> System.currentTimeMillis() - start;
return F.Promise.sequence(WS.url("http://www.google.com").get().map(timing),
WS.url("http://www.yahoo.com").get().map(timing),
WS.url("http://www.bing.com").get().map(timing))
.map(list -> {
Map<String, Long> data = new HashMap<>();
data.put("google", list.get(0));
data.put("yahoo", list.get(1));
data.put("bing", list.get(2));
return data;
})
.map(Json::toJson)
.map(Results::ok);
}
}
Finally, since Play 2.4 requires Java 8, this is a good opportunity to play around with lambdas!