We Keep Coding

Removing a property in JSON with Jackson?

In a Java/Spring application, I have a POJO/bean. Let's call it CourseRequest. It is converted/serialized into a string and sent out in a PUT REST request. The conversion is done using org.codehaus.jackson.map.ObjectMapper. The sending is done using org.scribe.model.OAuthRequest. See simplified code:
OAuthRequest request = new OAuthRequest(...);
CourseRequest courseRequest = new CourseRequest(...);
ObjectMapper mapper = new ObjectMapper();
String courseJson = null;
try {
// convert to json.
courseJson = mapper.writeValueAsString(courseRequest);
request.addPayload(courseJson);
request.send();
} catch (JsonGenerationException e) {
logger.error(e);
} catch (JsonMappingException e) {
logger.error(e);
} catch (IOException e) {
logger.error(e);
}
I have run into a problem where I need to remove a property instructorId from the JSON/payload before sending the request. Obviously, I can't just naively remove the property in CourseRequest because the POJO/bean is used in other places too.
So, what is the best way to do it?
I can't use #JsonIgnore or alike annotations, because
I want to remove the property only on some conditions;
There are other requests elsewhere in the application which may want to keep the property
For initial thoughts:
As a "hack", I can do a regex replace on the courseJson to remove the property after mapper.writeValueAsString(courseRequest) but I think that's not a very clean way to do it.
I can parse the courseJson back into some kind of map and then remove the property but that's just clumsy.
P.S. I am using Jackson v1.9.13

How to save objects in GridFS without taking them from a file?

I'm working in a Spring Boot api that can receive very large objects and try to save it in a MongoDB database. Because of this the program sometimes throws me the next error:
org.bson.BsonMaximumSizeExceededException: Payload document size is larger than maximum of 16793600.
I'd read that MongoDB only permits objects of size below 16MB, this is very inconvenient for my system because an object can easily surpass this gap. To solve this I had read about GridFS, technology that allows to surpass the 16MB files gap.
Now I'm trying to implement GridFS in my system but I only had seen examples using files to save in the database, something like this:
gridFsOperations.store(new FileInputStream("/Users/myuser/Desktop/text.txt"), "myText.txt", "text/plain", metaData);
But I want to do is not to take the data from a file, but to the api to receive a object and save it, something like this:
#PostMapping
public String save(#RequestBody Object object){
DBObject metaData = new BasicDBObject();
metaData.put("type", "data");
gridFsOperations.store(object, metaData);
return "Stored successfully...";
}
Is it a posible way to doing this?

Get an InputStream from the request and pass it to a GridFSBucket. Here's a rough example:
In your controller:
#PostMapping
public ResponseEntity<String> uploadFile(MultipartHttpServletRequest request)
{
Iterator<String> iterator = request.getFilenames();
String filename = iterator.next();
MultipartFile mf = request.getFile(filename);
// I always have a service layer between controller and repository but for purposes of this example...
myDao.uploadFile(filename, mf.getInputStream());
}
In your DAO/repository:
private GridFSBucket bucket;
#Autowired
void setMongoDatabase(MongoDatabase db)
{
bucket = GridFSBuckets.create(db);
}
public ObjectId uploadFile(String filename, InputStream is)
{
Document metadata = new Document("type", "data");
GridFSUploadOptions opts = new GridFSUploadOptions().metadata(metadata);
ObjectId oid = bucket.uploadFromStream(filename, is, opts);
try
{
is.close();
}
catch (IOException ioe)
{
throw new UncheckedIOException(ioe);
}
return oid;
}
I paraphrased this from existing code so it may not be perfect but will be good enough to point you in the right direction.

How do I send nested json POST request in java using jersey?

I am using a document converter api called cloudconvert. They don't have an official java library, but a third party java option. I needed a little customization so I cloned the github project and added it to my project. I am sending cloudconvert a .epub file and getting a .pdf file in return. If I use the default settings it works without issue and properly converts my .epub to a .pdf. Here is the code that makes it happen.
Here is what triggers the conversion:
// Create service object
CloudConvertService service = new CloudConvertService("api-key");
// Create conversion process
ConvertProcess process = service.startProcess(convertFrom, convertTo);
// Perform conversion
//convertFromFile is a File object with a .epub extension
process.startConversion(convertFromFile);
// Wait for result
ProcessStatus status;
waitLoop:
while (true) {
status = process.getStatus();
switch (status.step) {
case FINISHED:
break waitLoop;
case ERROR:
throw new RuntimeException(status.message);
}
// Be gentle
Thread.sleep(200);
}
//Download result
service.download(status.output.url, convertToFile);
//lean up
process.delete();
startConversion() calls:
public void startConversion(File file) throws ParseException, FileNotFoundException, IOException {
if (!file.exists()) {
throw new FileNotFoundException("File not found: " + file);
}
startConversion(new FileDataBodyPart("file", file));
}
Which calls this to actually send the POST request using jersey:
private void startConversion(BodyPart bodyPart) {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.bodyPart(bodyPart);
//root is a class level WebTarget object
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Up to this point everything is working. My problem is that the when the conversion happens the .pdf that returns has very small margins. cloudconvert provides a way to change those margins. You can send in an optional json param converteroptions and set the margins manually. I have tested this out using postman and it works without issue, I was able to get a properly formatted margin document. So know this is possible. Here is the POSTMAN info I used:
#POST : https://host123d1qo.cloudconvert.com/process/WDK9Yq0z1xso6ETgvpVQ
Headers: 'Content-Type' : 'application/json'
Body:
{
"input": "base64",
"file": "0AwAAIhMAAAAA", //base64 file string that is much longer than this
"outputformat": "pdf",
"converteroptions": {
"margin_bottom": 75,
"margin_top": 75,
"margin_right": 50,
"margin_left": 50
}
}
Here are my attempts at getting the POST request formatted properly, I'm just not very experienced with jersey and the couple of answers I did find on stackoverflow didn't work for me.
Attempt 1, I tried adding the json string as a Multipart.field. It didn't give me any errors and still returned a converted .pdf file, but the margins didn't get changed so I must not be sending it back right.
private void startConversion(BodyPart bodyPart) {
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.field("converteroptions", jsonString)
.bodyPart(bodyPart);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 2, when I had it working in POSTMAN it was using the 'input' type as 'base64' so I tried changing it to that but it this time it doesn't return anything at all, no request errors, just a timeout error at the 5 minute mark.
//I pass in a File object rather than the bodypart object.
private void startConversion(File file) {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "base64")
.field("outputformat", args.outputformat)
.field("file", encoded64)
.field("converteroptions", jsonString);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 3, after some googling on how to properly send jersey json post requests I changed the format. This time it returned a 400 bad request error.
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"input\":\"base64\",\"file\":\"" + encoded64 + "\",\"outputformat\":\"pdf\",\"converteroptions\":{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}}";
root.request(MediaType.APPLICATION_JSON).post(Entity.json(jsonString));
}
Attempt 4, Someone said you don't need to manually use a jsonString you should use serializable java beans. So I created the corresponding classes and made the request like shown below. Same 400 bad request error.
#XmlRootElement
public class PDFConvert implements Serializable {
private String input;
private String file;
private String outputformat;
private ConverterOptions converteroptions;
//with the a default constructor and getters/setters for all
}
#XmlRootElement
public class ConverterOptions implements Serializable {
private int margin_bottom;
private int margin_top;
private int margin_left;
private int margin_right;
//with the a default constructor and getters/setters for all
}
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
PDFConvert data = new PDFConvert();
data.setInput("base64");
data.setFile(encoded64);
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(75);
converteroptions.setMargin_bottom(75);
converteroptions.setMargin_left(50);
converteroptions.setMargin_right(50);
data.setConverteroptions(converteroptions);
root.request(MediaType.APPLICATION_JSON).post(Entity.json(data));
}
I know this is quite the wall of text, but I wanted to show all the different things I tried so that I wouldn't waste anyone's time. Thank you for any help or ideas you might have to make this work. I really want to make it work with jersey because I have several other conversions I do that work perfectly, they just don't need any converteroptions. Also I know its possible because it works when manually running the process through POSTMAN.
Cloudconvert api documentation for starting a conversion
Github repo with the recommended 3rd party java library I am using/modifying

I finally figured it out. Hours of trial and error. Here is the code that did it:
private void startConversionPDF(File file) throws IOException {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
PDFConvert data = new PDFConvert();
data.setInput("upload");
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(60);
converteroptions.setMargin_bottom(60);
converteroptions.setMargin_left(30);
converteroptions.setMargin_right(30);
data.setConverteroptions(converteroptions);
MultiPart multipart = new FormDataMultiPart()
.bodyPart(new FormDataBodyPart("json", data, MediaType.APPLICATION_JSON_TYPE))
.bodyPart(new FileDataBodyPart("file", file));
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}

Mocking/Testing HTTP Get Request

I'm trying to write unit tests for my program and use mock data. I'm a little confused on how to intercept an HTTP Get request to a URL.
My program calls a URL to our API and it is returned a simple XML file. I would like the test to instead of getting the XML file from the API online to receive a predetermined XML file from me so that I can compare the output to the expected output and determine if everything is working correctly.
I was pointed to Mockito and have been seeing many different examples such as this SO post, How to use mockito for testing a REST service? but it's not becoming clear to me how to set it all up and how to mock the data (i.e., return my own xml file whenever the call to the URL is made).
The only thing I can think of is having another program made that's running locally on Tomcat and in my test pass a special URL that calls the locally running program on Tomcat and then return the xml file that I want to test with. But that just seems like overkill and I don't think that would be acceptable. Could someone please point me in the right direction.
private static InputStream getContent(String uri) {
HttpURLConnection connection = null;
try {
URL url = new URL(uri);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/xml");
return connection.getInputStream();
} catch (MalformedURLException e) {
LOGGER.error("internal error", e);
} catch (IOException e) {
LOGGER.error("internal error", e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
return null;
}
I am using Spring Boot and other parts of the Spring Framework if that helps.

Part of the problem is that you're not breaking things down into interfaces. You need to wrap getContent into an interface and provide a concrete class implementing the interface. This concrete class will then
need to be passed into any class that uses the original getContent. (This is essentially dependency inversion.) Your code will end up looking something like this.
public interface IUrlStreamSource {
InputStream getContent(String uri)
}
public class SimpleUrlStreamSource implements IUrlStreamSource {
protected final Logger LOGGER;
public SimpleUrlStreamSource(Logger LOGGER) {
this.LOGGER = LOGGER;
}
// pulled out to allow test classes to provide
// a version that returns mock objects
protected URL stringToUrl(String uri) throws MalformedURLException {
return new URL(uri);
}
public InputStream getContent(String uri) {
HttpURLConnection connection = null;
try {
Url url = stringToUrl(uri);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/xml");
return connection.getInputStream();
} catch (MalformedURLException e) {
LOGGER.error("internal error", e);
} catch (IOException e) {
LOGGER.error("internal error", e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
return null;
}
}
Now code that was using the static getContent should go through a IUrlStreamSource instances getContent(). You then provide to the object that you want to test a mocked IUrlStreamSource rather than a SimpleUrlStreamSource.
If you want to test SimpleUrlStreamSource (but there's not much to test), then you can create a derived class that provides an implementation of stringToUrl that returns a mock (or throws an exception).

The other answers in here advise you to refactor your code to using a sort of provider which you can replace during your tests - which is the better approach.
If that isn't a possibility for whatever reason you can install a custom URLStreamHandlerFactory that intercepts the URLs you want to "mock" and falls back to the standard implementation for URLs that shouldn't be intercepted.
Note that this is irreversible, so you can't remove the InterceptingUrlStreamHandlerFactory once it's installed - the only way to get rid of it is to restart the JVM. You could implement a flag in it to disable it and return null for all lookups - which would produce the same results.
URLInterceptionDemo.java:
public class URLInterceptionDemo {
private static final String INTERCEPT_HOST = "dummy-host.com";
public static void main(String[] args) throws IOException {
// Install our own stream handler factory
URL.setURLStreamHandlerFactory(new InterceptingUrlStreamHandlerFactory());
// Fetch an intercepted URL
printUrlContents(new URL("http://dummy-host.com/message.txt"));
// Fetch another URL that shouldn't be intercepted
printUrlContents(new URL("http://httpbin.org/user-agent"));
}
private static void printUrlContents(URL url) throws IOException {
try(InputStream stream = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
String line;
while((line = reader.readLine()) != null) {
System.out.println(line);
}
}
}
private static class InterceptingUrlStreamHandlerFactory implements URLStreamHandlerFactory {
#Override
public URLStreamHandler createURLStreamHandler(final String protocol) {
if("http".equalsIgnoreCase(protocol)) {
// Intercept HTTP requests
return new InterceptingHttpUrlStreamHandler();
}
return null;
}
}
private static class InterceptingHttpUrlStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(final URL u) throws IOException {
if(INTERCEPT_HOST.equals(u.getHost())) {
// This URL should be intercepted, return the file from the classpath
return URLInterceptionDemo.class.getResource(u.getHost() + "/" + u.getPath()).openConnection();
}
// Fall back to the default handler, by passing the default handler here we won't end up
// in the factory again - which would trigger infinite recursion
return new URL(null, u.toString(), new sun.net.www.protocol.http.Handler()).openConnection();
}
}
}
dummy-host.com/message.txt:
Hello World!
When run, this app will output:
Hello World!
{
"user-agent": "Java/1.8.0_45"
}
It's pretty easy to change the criteria of how you decide which URLs to intercept and what you return instead.

The answer depends on what you are testing.
If you need to test the processing of the InputStream
If getContent() is called by some code that processes the data returned by the InputStream, and you want to test how the processing code handles specific sets of input, then you need to create a seam to enable testing. I would simply move getContent() into a new class, and inject that class into the class that does the processing:
public interface ContentSource {
InputStream getContent(String uri);
}
You could create a HttpContentSource that uses URL.openConnection() (or, better yet, the Apache HttpClientcode).
Then you would inject the ContentSource into the processor:
public class Processor {
private final ContentSource contentSource;
#Inject
public Processor(ContentSource contentSource) {
this.contentSource = contentSource;
}
...
}
The code in Processor could be tested with a mock ContentSource.
If you need to test the fetching of the content
If you want to make sure that getContent() works, you could create a test that starts a lightweight in-memory HTTP server that serves the expected content, and have getContent() talk to that server. That does seem overkill.
If you need to test a large subset of the system with fake data
If you want to make sure things work end to end, write an end to end system test. Since you indicated you use Spring, you can use Spring to wire together parts of the system (or to wire the entire system, but with different properties). You have two choices
Have the system test start a local HTTP server, and when you have your test create your system, configure it to talk to that server. See the answers to this question for ways to start the HTTP server.
Configure spring to use a fake implementation of ContentSource. This gets you slightly less confidence that everything works end-to-end, but it will be faster and less flaky.

webservice client implementaion in axis2-1.6.2 in java

I have implemented webservice client in axis2-1.6.2 in java and I get response when I call first time and for subsequent second time I get below error
java.lang.NullPointerException
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:293)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:180)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.apache.axis2.ccws.CustomerCareServiceStub.subscriberRetrieveLite(CustomerCareServiceStub.java:2380)
at Prepost.SubscriberRetrieveBalance.subscriberRetrieveLite(SubscriberRetrieveBalance.java:111)
at Prepost.CheckUser.doGet(CheckUser.java:149)
here is the API implementation class constructor which sets unique parameter which is same for all requests
public SubscriberRetrieveBalance(String url, String strCON_TimeOut, String strSO_TimeOut) {
try {
this.url = url;
stub = new CustomerCareServiceStub(url);
ServiceClient sClient = stub._getServiceClient();
Options options = sClient.getOptions();
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
options.setProperty(AddressingConstants.WS_ADDRESSING_VERSION, AddressingConstants.Submission.WSA_NAMESPACE);
//options.setTimeOutInMilliSeconds(2000);
TransportInDescription transportIn = new TransportInDescription("HTTP");
options.setTransportIn(transportIn);
options.setProperty(HTTPConstants.SO_TIMEOUT, Integer.parseInt(strSO_TimeOut));
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, Integer.parseInt(strCON_TimeOut));
sClient.setOptions(options);
} catch (Exception e) {
if (e.getMessage().equals("Can not find the external id")) {
System.out.println("Exception ::" + e.getMessage());
}
}
}
and it is called in a servlet and for performance issue I make object of this class for different-2 states(urls) and saved these object to hashmap when first request comes for respective states then make new object and use that object for subsequent requests for that state
SubscriberRetrieveBalance objBal = null;
BalanceBean bal = new BalanceBean();
if (mapObj.isEmpty() || (mapObj.get(strIP) == null)) {
objBal = new SubscriberRetrieveBalance(url, strCON_TimeOut, strSO_TimeOut);
mapObj.put(strIP, objBal);
} else {
objBal = mapObj.get(strIP);
}
bal = objBal.subscriberRetrieveLite(strMsisdn, userId, token, strCircleId, strCircleName, strSessionId, strDlgId);
first time it gives response and then gives nullpointer exception and above error for all requests that belongs to that state
This code is working fine with axis2-1.5
Is there any change in axis2-1.6.2 version that every time it needs new object of API implemented class
Please suggest.