We Keep Coding

How to save objects in GridFS without taking them from a file?

I'm working in a Spring Boot api that can receive very large objects and try to save it in a MongoDB database. Because of this the program sometimes throws me the next error:
org.bson.BsonMaximumSizeExceededException: Payload document size is larger than maximum of 16793600.
I'd read that MongoDB only permits objects of size below 16MB, this is very inconvenient for my system because an object can easily surpass this gap. To solve this I had read about GridFS, technology that allows to surpass the 16MB files gap.
Now I'm trying to implement GridFS in my system but I only had seen examples using files to save in the database, something like this:
gridFsOperations.store(new FileInputStream("/Users/myuser/Desktop/text.txt"), "myText.txt", "text/plain", metaData);
But I want to do is not to take the data from a file, but to the api to receive a object and save it, something like this:
#PostMapping
public String save(#RequestBody Object object){
DBObject metaData = new BasicDBObject();
metaData.put("type", "data");
gridFsOperations.store(object, metaData);
return "Stored successfully...";
}
Is it a posible way to doing this?

Get an InputStream from the request and pass it to a GridFSBucket. Here's a rough example:
In your controller:
#PostMapping
public ResponseEntity<String> uploadFile(MultipartHttpServletRequest request)
{
Iterator<String> iterator = request.getFilenames();
String filename = iterator.next();
MultipartFile mf = request.getFile(filename);
// I always have a service layer between controller and repository but for purposes of this example...
myDao.uploadFile(filename, mf.getInputStream());
}
In your DAO/repository:
private GridFSBucket bucket;
#Autowired
void setMongoDatabase(MongoDatabase db)
{
bucket = GridFSBuckets.create(db);
}
public ObjectId uploadFile(String filename, InputStream is)
{
Document metadata = new Document("type", "data");
GridFSUploadOptions opts = new GridFSUploadOptions().metadata(metadata);
ObjectId oid = bucket.uploadFromStream(filename, is, opts);
try
{
is.close();
}
catch (IOException ioe)
{
throw new UncheckedIOException(ioe);
}
return oid;
}
I paraphrased this from existing code so it may not be perfect but will be good enough to point you in the right direction.

Offline XML Validation with Java

I need to figure out how to validate my XML files with schema's offline. After looking around for a couple of days, what I was able to find was basically that I needed to have an internal reference to the schema. I needed to find them, download them, and change the reference to a local system path. What I was unable to find was exactly how to do that. Where and how can I change the reference to point internally instead of externally? What is the best way to download the schemas?

There are three ways you could do this. What they all have in common is that you need a local copy of the schema document(s). I'm assuming that the instance documents currently use xsi:schemaLocation and/or xsi:noNamespaceSchemaLocation to point to a location holding the schema document(s) on the web.
(a) Modify your instance documents to refer to the local copy of the schema documents. This is usually inconvenient.
(b) Redirect the references so that a request for a remote file is redirected to a local file. The way to set this up depends on which schema validator you are using and how you are invoking it.
(c) Tell the schema processor to ignore the values of xsi:schemaLocation and xsi:noNamespaceSchemaLocation, and to validate instead against a schema that you supply using your schema processor's invocation API. Again the details depend on which schema processor you are using.
My preferred approach is (c): if only because when you are validating a source document, then by definition you don't fully trust it - so why should you trust it to contain a correct xsi:schemaLocation attribute?

XmlValidate is a simple but powerful command-line tool that can perform offline validation of single or multiple XML files against target schemas. It can scan local xml files by file name, directory, or URL.
XmlValidate automatically adds the schemaLocation based on the schema namespace and a config file that mapping to a local file. The tool will validate against whatever XML Schema is referenced in the config file.
Here are example mappings of namespace to target Schema in config file:
http://www.opengis.net/kml/2.2=${XV_HOME}/schemas/kml22.xsd
http://appengine.google.com/ns/1.0=C:/xml/appengine-web.xsd
urn:oasis:names:tc:ciq:xsdschema:xAL:2.0=C:/xml/xAL.xsd
Note that ${XV_HOME} token above is simply an alias for the top-level directory that XmlValidate is running from. The location can likewise be a full file path.
XmlValidate is an open-source project (source code available) that runs with the Java Runtime Environment (JRE). The bundled application (Java jars, examples, etc.) can be downloaded here.
If XmlValidate is run in batch mode against multiple XML files, it will provide a summary of validation results.
Errors: 17 Warnings: 0 Files: 11 Time: 1506 ms
Valid files 8/11 (73%)

You can set your own Implementation of ResourceResolver and LSInput to the SchemaFactory so that the call of
of LSInput.getCharacterStream() will provide a schema from a local path.
I have written an extra class to do offline validation. You can call it like
new XmlSchemaValidator().validate(xmlStream, schemaStream, "https://schema.datacite.org/meta/kernel-4.1/",
"schemas/datacite/kernel-4.1/");
Two InputStream are beeing passed. One for the xml, one for the schema. A baseUrl and a localPath (relative on classpath) are passed as third and fourth parameter. The last two parameters are used by the validator to lookup additional schemas locally at localPath or relative to the provided baseUrl.
I have tested with a set of schemas and examples from https://schema.datacite.org/meta/kernel-4.1/ .
Complete Example:
#Test
public void validate4() throws Exception {
InputStream xmlStream = Thread.currentThread().getContextClassLoader().getResourceAsStream(
"schemas/datacite/kernel-4.1/example/datacite-example-complicated-v4.1.xml");
InputStream schemaStream = Thread.currentThread().getContextClassLoader()
.getResourceAsStream("schemas/datacite/kernel-4.1/metadata.xsd");
new XmlSchemaValidator().validate(xmlStream, schemaStream, "https://schema.datacite.org/meta/kernel-4.1/",
"schemas/datacite/kernel-4.1/");
}
The XmlSchemaValidator will validate the xml against the schema and will search locally for included Schemas. It uses a ResourceResolver to override the standard behaviour and to search locally.
public class XmlSchemaValidator {
/**
* #param xmlStream
* xml data as a stream
* #param schemaStream
* schema as a stream
* #param baseUri
* to search for relative pathes on the web
* #param localPath
* to search for schemas on a local directory
* #throws SAXException
* if validation fails
* #throws IOException
* not further specified
*/
public void validate(InputStream xmlStream, InputStream schemaStream, String baseUri, String localPath)
throws SAXException, IOException {
Source xmlFile = new StreamSource(xmlStream);
SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
factory.setResourceResolver((type, namespaceURI, publicId, systemId, baseURI) -> {
LSInput input = new DOMInputImpl();
input.setPublicId(publicId);
input.setSystemId(systemId);
input.setBaseURI(baseUri);
input.setCharacterStream(new InputStreamReader(
getSchemaAsStream(input.getSystemId(), input.getBaseURI(), localPath)));
return input;
});
Schema schema = factory.newSchema(new StreamSource(schemaStream));
javax.xml.validation.Validator validator = schema.newValidator();
validator.validate(xmlFile);
}
private InputStream getSchemaAsStream(String systemId, String baseUri, String localPath) {
InputStream in = getSchemaFromClasspath(systemId, localPath);
// You could just return in; , if you are sure that everything is on
// your machine. Here I call getSchemaFromWeb as last resort.
return in == null ? getSchemaFromWeb(baseUri, systemId) : in;
}
private InputStream getSchemaFromClasspath(String systemId, String localPath) {
System.out.println("Try to get stuff from localdir: " + localPath + systemId);
return Thread.currentThread().getContextClassLoader().getResourceAsStream(localPath + systemId);
}
/*
* You can leave out the webstuff if you are sure that everything is
* available on your machine
*/
private InputStream getSchemaFromWeb(String baseUri, String systemId) {
try {
URI uri = new URI(systemId);
if (uri.isAbsolute()) {
System.out.println("Get stuff from web: " + systemId);
return urlToInputStream(uri.toURL(), "text/xml");
}
System.out.println("Get stuff from web: Host: " + baseUri + " Path: " + systemId);
return getSchemaRelativeToBaseUri(baseUri, systemId);
} catch (Exception e) {
// maybe the systemId is not a valid URI or
// the web has nothing to offer under this address
}
return null;
}
private InputStream urlToInputStream(URL url, String accept) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setRequestProperty("User-Agent", "Name of my application.");
con.setReadTimeout(15000);
con.setRequestProperty("Accept", accept);
con.connect();
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_MOVED_PERM
|| responseCode == HttpURLConnection.HTTP_MOVED_TEMP || responseCode == 307
|| responseCode == 303) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, accept);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, accept);
}
}
inputStream = con.getInputStream();
return inputStream;
} catch (SocketTimeoutException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private InputStream getSchemaRelativeToBaseUri(String baseUri, String systemId) {
try {
URL url = new URL(baseUri + systemId);
return urlToInputStream(url, "text/xml");
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
prints
Try to get stuff from localdir: schemas/datacite/kernel-4.1/http://www.w3.org/2009/01/xml.xsd
Get stuff from web: http://www.w3.org/2009/01/xml.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-titleType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-contributorType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-dateType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-resourceType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-relationType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-relatedIdentifierType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-funderIdentifierType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-descriptionType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-nameType-v4.1.xsd
The print shows that the validator was able to validate against a set of local schemas. Only http://www.w3.org/2009/01/xml.xsd was not available locally and therefore fetched from the internet.

How do I validate an XML file against an XSD through https URL?

I would like to validate an XML file using a schema located at a secure https site. How do I tell the validator to except a self-signed certificate or use an https URL? I have a file called test.xml and a schema located at https://localhost:1234/module/testschema.xsd. I'm using the same code found here. If I use a regular URL (http://localhost/module/testschema.xsd), it works great. If I substitute with an https URL, then I get this error:
schema_reference.4: Failed to read schema document 'https://localhost:1234/module/testschema.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
Copied Code:
public boolean validateFile(String xml, String strSchemaLocation)
{
Source xmlFile = null;
try {
URL schemaFile = new URL(strSchemaLocation);
xmlFile = new StreamSource(new File(xml));
SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(schemaFile);
Validator validator = schema.newValidator();
validator.validate(xmlFile);
System.out.println(xmlFile.getSystemId() + " is valid");
} catch (SAXException e) {
System.out.println(xmlFile.getSystemId() + " is NOT valid");
System.out.println("Reason: " + e.getLocalizedMessage());
return false;
} catch (IOException ioe) {
System.out.println("IOException");
return false;
}
return true;
}

This has very little to to do with schema validation. Your problem is that you need to establish an HTTPS connection and trust a self-signed certificate. See How can I use different certificates on specific connections? or google around for that.
I don't think you'll be able to use the SchemaFactory.newSchema factory method that takes a File, so just use the one that takes a StreamSource:
URL schemaFile = new URL(strSchemaLocation);
HttpsURLConnection schemaConn = (HttpsURLConnection)schemaFile.openConnection();
// Magic from the other answer to accept self-signed cert
InputStream is = schemaConn.getInputStream();
SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(new StreamSource(is));
(I'm leaving out the try..catch to close the input stream and the connection)

It's not a validation problem, java.net.URL supports https, there should be bo difference. Just make sure that you can open https://localhost:1234/module/testschema.xsd with a browser.

Consume SAMLResponse Token

SAML sp-based authentication has following short workflow.
User wants to access application at sp.
sp sends SAMLRequest token to idp.
idp consume it and generate SAMLResponse token.
idp sends this SAMLResponse token to AC-URL given by sp.
My Question is how sp consume this SAMLResponse token.
What is the logic?
If I can get some JAVA code help it will be beneficial.

The next recipe is working for me:
Get the SAMLResponse token and decode it and inflate:
// Base64 decode
Base64 base64Decoder = new Base64();
byte[] xmlBytes = encodedXmlString.getBytes("UTF-8");
byte[] base64DecodedByteArray = base64Decoder.decode(xmlBytes);
// Inflate (uncompress) the AuthnRequest data
// First attempt to unzip the byte array according to DEFLATE (rfc 1951)
Inflater inflater = new Inflater(true);
inflater.setInput(base64DecodedByteArray);
// since we are decompressing, it's impossible to know how much space we
// might need; hopefully this number is suitably big
byte[] xmlMessageBytes = new byte[5000];
int resultLength = inflater.inflate(xmlMessageBytes);
if (!inflater.finished()) {
throw new RuntimeException("didn't allocate enough space to hold "
+ "decompressed data");
}
inflater.end();
String decodedResponse = new String(xmlMessageBytes, 0, resultLength,
"UTF-8");
return decodedResponse;
Parse the resulting XML. Here you can get the info that you need and for example, create a POJO with it (this is a sample code for parsing LogoutRequest's but would be analogous for responses):
// Parse the XML. SAX approach, we just need the ID attribute
SAXParserFactory saxParserFactory = SAXParserFactory.newInstance();
// If we want to validate the doc we need to load the DTD
// saxParserFactory.setValidating(true);
// Get a SAXParser instance
SAXParser saxParser = saxParserFactory.newSAXParser();
// Parse it
XMLhandler xmLhandler = new XMLhandler();
saxParser.parse(new ByteArrayInputStream(xmlLogoutRequest.getBytes()),
xmLhandler);
// Return the SamlVO
return xmLhandler.getSamlVO();
For my use case I am interesting in only a few elements, so I am using SAX:
public class XMLhandler extends DefaultHandler {
private SamlVO samlVO;
public XMLhandler() {
samlVO = new SamlVO();
}
#Override
public void startElement(String uri, String localName, String qName,
Attributes attributes) throws SAXException {
// Managing a LogoutRequest means that we are going to build a LogoutResponse
if (qName.equals("samlp:LogoutRequest")) {
// The ID value of a request will be the LogoutResponse's InReponseTo attribute
samlVO.setInResponseTo(attributes.getValue("ID"));
// From the destination we can get the Issuer element
String destination = attributes.getValue("Destination");
if (destination != null) {
URL destinationUrl = null;
try {
destinationUrl = new URL(destination);
} catch (MalformedURLException e) {
// TODO: We could set the server hostname (take it from a property), but this URL SHOULD be well formed!
e.printStackTrace();
}
samlVO.setIssuer(destinationUrl.getHost());
}
}
}
public SamlVO getSamlVO() {
return samlVO;
}
}
Hope it helps,
Luis
PS: you also can use a library like OpenSAML
DefaultBootstrap.bootstrap();
HTTPRedirectDeflateDecoder decode = new HTTPRedirectDeflateDecoder(new BasicParserPool());
BasicSAMLMessageContext<LogoutRequest, ?, ?> messageContext = new BasicSAMLMessageContext<LogoutRequest, SAMLObject, SAMLObject>();
messageContext.setInboundMessageTransport(new HttpServletRequestAdapter(request));
decode.decode(messageContext);
XMLObjectBuilderFactory builderFactory = org.opensaml.Configuration.getBuilderFactory();
LogoutRequestBuilder logoutRequestBuilder = (LogoutRequestBuilder) builderFactory.getBuilder(LogoutRequest.DEFAULT_ELEMENT_NAME);
LogoutRequest logoutRequest = logoutRequestBuilder.buildObject();
logoutRequest = (LogoutRequest) messageContext.getInboundMessage();
But be prepared to include a few libraries in your CLASSPATH!!!

Here is how I do it in Java. I use XMLBeans to parse the SAMLResponse, then decrypt it (if it's encrypted) and then verify the signature:
WebBrowserSSOAuthConsumerService

Asking for code is a bit much, but the basic processing is that the SP validates the SAMLResponse, including for well-formedness, presence of required values, correct protocol, and any other SP-specific validation (time constraints, data correspondence, etc.), maps user identified in token to user on SP (could involve creating user), and transfers user to requested resource.

using freemarker and spring to construct templates

I am new to freemarker. I have a spring application that I am planning to use with freemarker. Templates will be stored in database and based on the login, I want to retrieve the template from database. Can any one tell me how to configure the freemarker in spring and get the html tags as a string after constructing the template. I did googling but I could not understand much.
I tried till this level. In spring I have done till this level. Finally I want html tags in a string.
// Spring freemarker specific code
Configuration configuration = freemarkerConfig.getConfiguration();
StringTemplateLoader stringTemplateLoader = new StringTemplateLoader();
// My application specific code
String temp = tempLoader.getTemplateForCurrentLogin();
Thanks.

To tie together the bits of code you posted, you can do something like this:
// you already have this bit
String templateText = tempLoader.getTemplateForCurrentLogin();
// now programmatically instantiate a template
Template t = new Template("t", new StringReader(templateText), new Configuration());
// now use the Spring utility class to process it into a string
// myData is your data model
String output = FreeMarkerTemplateUtils.processTemplateIntoString(template, myData);

This java method will process the freemarker template and will give html tags as String after constructing the template.
public static String processFreemarkerTemplate(String fileName) {
StringWriter stringWriter = new StringWriter();
Map<String, Object> objectMap = new HashMap<>();
Configuration cfg = new Configuration(Configuration.VERSION_2_3_24);
try {
cfg.setDirectoryForTemplateLoading(new File("path/of/freemarker/template"));
cfg.setDefaultEncoding("UTF-8");
cfg.setTemplateExceptionHandler(TemplateExceptionHandler.RETHROW_HANDLER);
cfg.setLogTemplateExceptions(false);
Template template = cfg.getTemplate(fileName);
template.process(objectMap, stringWriter);
} catch (IOException | TemplateException e) {
e.printStackTrace();
} finally {
if (stringWriter != null) {
try {
stringWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return stringWriter.toString();
}