I just started apache-spark with java. There is many documents saved in a collection I want to find a document on the basis of some key and update it.
Simply I want to do find and update in apache-spark with java
This is the code to read a document form mongo
Map<String, String> readOverrides = new HashMap<String, String>();
readOverrides.put("collection", "bb_playing22");
ReadConfig readConfig = ReadConfig.create(createJavaSparkContext()).withOptions(readOverrides);
JavaMongoRDD<Document> customRdd = MongoSpark.load(createJavaSparkContext(), readConfig);
JavaRDD<Document> rdd = customRdd.filter(((t1) -> {
return t1.getLong("playing22_id") == 3; //To change body of generated lambdas, choose Tools | Templates.
}));
but not able to update this document
Spark does a lazy evaluation of your transformations hence unless you call an action over an rdd, it won't execute any transformations you code.
See common actions here.
Also, this does not seem like a use case where you'd like to use spark.
If your actual requirement is just to update a document as per a value then you should look into indexing in mongo and just update a document through a mongo driver in java directly.
Related
I've put a HashMap<String, Set<Long>> object into a MongoDB document under "disabled_channels" but I can't figure out how to retrieve it and turn it back into a HashMap<String, Set<Long>> object in local memory. I'm usually very good at reading in lists, individual values, etc, with something like found.getList("disabled_commands", String.class) but I'm really lost on how to approach this.
MongoCollection<Document> collection = bot.getDataManager().getConfig();
Document config = new Document("guild", guild.getIdLong());
Document found = collection.find(config).first();
// I get lost here
Document itself is a map implementation internally. Reference
You need to use get function on found document and cast it to Document as below
Document channels = (Document)found.get("disabled_channels")
Then you can access elements in channels using the same get method and cast it as per the need.
I have a scenario where in we get machine code from one machine that needs to be sent to another but by converting it to string which other machine understands following are the scenarios
**if code is 'AGO PRF' then convert to 'AGO.P'
if code is 'HUSQ A' then convert to 'HUSQ.A'
if code is 'AIK B' then convert to 'AIK.B'
if code is 'ALUS WS' then convert to 'ALUS.WS'
if code is 'VST WSA' then convert to 'VST.WSA'
if code is 'SAB CL' then convert to 'SAB.CL'
if code is 'SPR WSB' then convert to 'NSPR.WSB'
if code is 'AXS PRD CL' then change it to 'AXS.PCL'
if code is 'CTEST RT' then convert to 'CTEST.R'
if code is 'ALUS U' then convert to 'ALUS.U'
if code is 'SFUN WI' then convert to 'SFUN.WI'
if code is 'RQI RT WI' then convert to 'RQI.RTWI'
if code is 'ECA WS WI' then change it to 'ECA.WSWI'.**
I used a MAP to fed in these values as keys and give out the output. but I want to know if there can be more generic solution to this
If there exists neither a rule nor a regularity of the String replacement (I find none), then you need either a mapping table stored in the DB or a static Map<String, String> of these constants:
I recommend the Map in case of a small number of these and they would not be changed often.
I recommend reading from the DB in case of a larger number of these. This also allows you to change the mapping on run with no need to build and redeploy the entire application.
In terms of the data structure, the dictionary-based would be the best way to go - Map<String, String>. It doesn't allow you to store duplicated keys and is simple to use for the transformation:
List<String> listOfStringsToBeReplaced = loadFromSomewhere();
Map<String, String> map = loadFromDb();
List<String> listWithReplacedStrnigs = listOfStringsToBeReplaced.stream()
.map(string -> map.getOrDefault(string, string))
.collect(Collectors.toList());
I use Map::getOrDefault to either replace the value or keep it as is if no mapping is found.
I know how to tag single s3 object from my java app. But how can i tag multiple objects at once, if i pass multiple keys ?
We are talking about a big chunk of keys. ~200k
List<Tag> newTags = new ArrayList<Tag>();
newTags.add(new Tag("Tag for delete", "This object should be deleted"));
amazonS3Client.setObjectTagging(new SetObjectTaggingRequest(bucketName, key, new ObjectTagging(newTags)));
I am using the above code for single object tagging.
Yes, there is an option for batch tagging s3 objects in AWS.
Since you are asking for a way to do this, I recommend the s3 batch operations feature: https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/
You can create a manifest of s3 keys and provide the desired tags in the create job request, and it will tag all of the keys in the manifest with those tags.
I am fairly new to Freemarker and to template engine in general.
Basicly I want to be able to generate two XML files (from two different templates) using java methods to feed data to the template.
For example:
myTemplate1.ftl contain an opening and closing "res" tag with getRes() method as value (return 1 for exemple)
myTemplate2.ftl contain an opening and closing "result" tag and takes getResult() method as value (return 2 for exemple)
How can I write one class Main to process this without being specific (I do not want to write line per line, because then it would be pointless to create an engine)
Can you please help me understand how it work through an example if possible ?
Regards
EDIT with all the new informations :
#fustaki This is very frustrating. It seems I need to go step by step to understand what I'm doing. So... Here is a very dumb example of where I am :
template1.ftl
<Email>${item.getEmail()}</Email><Language>${item.getLanguage()}</Language>
FillMyTemplate1.java
public String getEmail(){ return "test#test.com" }
public String getLanguage(){ return "EN" }
I am using a property file in order to use introspection : prop.properties which contain :
template1=fr.freemarker.test.xml.FillMyTemplate1
MainTest.java
public static void main(String[] args) throws Exception {
Properties prop = new Properties();
InputStream input = new FileInputStream("prop.properties");
prop.load(input);
Class<?> classe = Class.forName(prop.getProperty(args[0])); //where args[0] is "template1"
Configuration config=new Configuration();
config.setClassForTemplateLoading(MainTest.class, "templates");
config.setObjectWrapper(new DefaultObjectWrapper());
Template template=config.getTemplate(args[0]+".ftl");
Map<String, Object> dataModel = new HashMap<String, Object>();
Writer consoleWriter = new OutputStreamWriter(System.out);
dataModel.put("item", classe.newInstance());
template.process(dataModel, consoleWriter);
}
And the result : <Email>test#test.com</Email><Language>EN</Language>
In order to understand what you said, I need to know how a "data-provider" would look like in my case ? How can I get ride of this "item" in my map ? I understand the principle but not the technical solution...
Your Main process is in charge of producing the data (List, Map or other structured object) and templates will use the data to render your XMLs.
The tags used in the templates are independent from the data inside them.
Example where data are stored in a List that can be retrieved with a getItems() or in general is accessible from the freemarker engine.
<?xml>
<res>
<#list items as item >
<item>${item}</item>
</#list>
</res>
You can create as much templates (views) as you want using the same List items.
I hope this clarify your troubles.
here is good example of freemarker. this application generate spring mvc crud operation
using a freemarker template.
springmvcfreemarker
I am really new to this big data i need to know can hbase be embedded in java application.
As is it developed in java can hbase be added as an library and do the operations?
If so can any one give an simple tutorial or sample code.
HBase does not run embedded it runs on top of Hadoop and it is aimed at big data and a large number of servers.
It does have a Java API which you can use e.g. Charles Menguy's reply
You can definitely write a Java application to use Hbase, there is a Java API that is provided in the main Hbase distribution.
You should take the latest build from the official website and get the hbase-0.xx.x.jar jar that you can use to build your application. If you want to look at classpath dependencies for hbase, once you installed hbase you can just do hbase classpath and that should print you the list of jars you need.
You can probably find a lot of example of Java apps doing Hbase operations on Google, but here is an example for the usual operations:
// get the hbase config
Configuration config = HBaseConfiguration.create();
// specify which table you want to use
HTable table = new HTable(config, "mytable");
// add a row to your table
Put p = new Put(Bytes.toBytes("myrow"));
// specify the column family, column qualifier and column value
p.add(Bytes.toBytes("myfamily"), Bytes.toBytes("myqualifier"), Bytes.toBytes("myvalue"));
// commit to your table
table.put(p);
// define which row you want to get
Get g = new Get(Bytes.toBytes("myrow"));
// get your row
Result r = table.get(g);
// choose what you want to extract from your row
byte[] value = r.getValue(Bytes.toBytes("myfamily"), Bytes.toBytes("myqualifier"));
// convert to a string
System.out.println("GET: " + Bytes.toString(value));
// do a scan operation
Scan s = new Scan();
s.addColumn(Bytes.toBytes("myfamily"), Bytes.toBytes("myqualifier"));
ResultScanner scanner = table.getScanner(s);