Parsing multiple sentences with MaltParser using NLTK - java

There have been many MaltParser and/or NLTK related questions:
Malt Parser throwing class not found exception
How to use malt parser in python nltk
MaltParser Not Working in Python NLTK
NLTK MaltParser won't parse
Dependency parser using NLTK and MaltParser
Dependency Parsing using MaltParser and NLTK
Parsing with MaltParser engmalt
Parse raw text with MaltParser in Java
Now, there's a more stabilized version of MaltParser API in NLTK: https://github.com/nltk/nltk/pull/944 but there are issues when it comes to parsing multiple sentences at the same time.
Parsing one sentence at a time seems fine:
_path_to_maltparser = '/home/alvas/maltparser-1.8/dist/maltparser-1.8/'
_path_to_model= '/home/alvas/engmalt.linear-1.7.mco'
>>> mp = MaltParser(path_to_maltparser=_path_to_maltparser, model=_path_to_model)
>>> sent = 'I shot an elephant in my pajamas'.split()
>>> sent2 = 'Time flies like banana'.split()
>>> print(mp.parse_one(sent).tree())
(pajamas (shot I) an elephant in my)
But parsing a list of sentences doesn't return a DependencyGraph object:
_path_to_maltparser = '/home/alvas/maltparser-1.8/dist/maltparser-1.8/'
_path_to_model= '/home/alvas/engmalt.linear-1.7.mco'
>>> mp = MaltParser(path_to_maltparser=_path_to_maltparser, model=_path_to_model)
>>> sent = 'I shot an elephant in my pajamas'.split()
>>> sent2 = 'Time flies like banana'.split()
>>> print(mp.parse_one(sent).tree())
(pajamas (shot I) an elephant in my)
>>> print(next(mp.parse_sents([sent,sent2])))
<listiterator object at 0x7f0a2e4d3d90>
>>> print(next(next(mp.parse_sents([sent,sent2]))))
[{u'address': 0,
u'ctag': u'TOP',
u'deps': [2],
u'feats': None,
u'lemma': None,
u'rel': u'TOP',
u'tag': u'TOP',
u'word': None},
{u'address': 1,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 2,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'I'},
{u'address': 2,
u'ctag': u'NN',
u'deps': [1, 11],
u'feats': u'_',
u'head': 0,
u'lemma': u'_',
u'rel': u'null',
u'tag': u'NN',
u'word': u'shot'},
{u'address': 3,
u'ctag': u'AT',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'AT',
u'word': u'an'},
{u'address': 4,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'elephant'},
{u'address': 5,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'in'},
{u'address': 6,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'my'},
{u'address': 7,
u'ctag': u'NNS',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NNS',
u'word': u'pajamas'},
{u'address': 8,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'Time'},
{u'address': 9,
u'ctag': u'NNS',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NNS',
u'word': u'flies'},
{u'address': 10,
u'ctag': u'NN',
u'deps': [],
u'feats': u'_',
u'head': 11,
u'lemma': u'_',
u'rel': u'nn',
u'tag': u'NN',
u'word': u'like'},
{u'address': 11,
u'ctag': u'NN',
u'deps': [3, 4, 5, 6, 7, 8, 9, 10],
u'feats': u'_',
u'head': 2,
u'lemma': u'_',
u'rel': u'dep',
u'tag': u'NN',
u'word': u'banana'}]
Why is that using parse_sents() don't return an iterable of parse_one?
I could however, just get lazy and do:
_path_to_maltparser = '/home/alvas/maltparser-1.8/dist/maltparser-1.8/'
_path_to_model= '/home/alvas/engmalt.linear-1.7.mco'
>>> mp = MaltParser(path_to_maltparser=_path_to_maltparser, model=_path_to_model)
>>> sent1 = 'I shot an elephant in my pajamas'.split()
>>> sent2 = 'Time flies like banana'.split()
>>> sentences = [sent1, sent2]
>>> for sent in sentences:
>>> ... print(mp.parse_one(sent).tree())
But this is not the solution I'm looking for. My question is how to answer why doesn't the parse_sent() return an iterable of parse_one(). and how could it be fixed in the NLTK code?
After #NikitaAstrakhantsev answered, I've tried it outputs a parse tree now but it seems to be confused and puts both sentences into one before parsing it.
# Initialize a MaltParser object with a pre-trained model.
mp = MaltParser(path_to_maltparser=path_to_maltparser, model=path_to_model)
sent = 'I shot an elephant in my pajamas'.split()
sent2 = 'Time flies like banana'.split()
# Parse a single sentence.
print(mp.parse_one(sent).tree())
print(next(next(mp.parse_sents([sent,sent2]))).tree())
[out]:
(pajamas (shot I) an elephant in my)
(shot I (banana an elephant in my pajamas Time flies like))
From the code it seems to be doing something weird: https://github.com/nltk/nltk/blob/develop/nltk/parse/api.py#L45
Why is it that the parser abstract class in NLTK is swooshing two sentences into one before parsing? Am I calling the parse_sents() incorrectly? If so, what is the correct way to call parse_sents()?


As I see in your code samples, you don't call tree() in this line
>>> print(next(next(mp.parse_sents([sent,sent2]))))
while you do call tree() in all cases with parse_one().
Otherwise I don't see the reason why it could happen: parse_one() method of ParserI isn't overridden in MaltParser and everything it does is simply calling parse_sents() of MaltParser, see the code.
Upd: The line you're talking about isn't called, because parse_sents() is overridden in MaltParser and is directly called.
The only guess I have now is that java lib maltparser doesn't work correctly with input file containing several sentences (I mean this block - where java is run). Maybe original malt parser has changed the format and now it is not '\n\n'.
Unfortunately, I can't run this code by myself, because maltparser.org is down for the second day. I checked that the input file has expected format (sentences are separated by double endline), so it is very unlikely that python wrapper merges sentences.

Related

Convert json to java object with gson

As part of my computer science IA I am creating a tool that reads match history and details of dota games and generates stats and hero stats. To do this I have accessed the valve API and grabbed a few jsons of matches and match history from it, then cut them down slightly so they only contain the information I need in the json.
Below is a sample of the details of one of the matches in a json format:
"result": {
"players": [
{
"account_id": 40884464,
"player_slot": 0,
"hero_id": 31,
"kills": 8,
"deaths": 8,
"assists": 14,
"last_hits": 72,
"denies": 0,
"gold_per_min": 304,
"xp_per_min": 412,
"level": 18,
},
{
"account_id": 70638797,
"player_slot": 1,
"hero_id": 35,
"kills": 6,
"deaths": 7,
"assists": 4,
"last_hits": 212,
"denies": 37,
"gold_per_min": 371,
"xp_per_min": 356,
"level": 17,
},
{
"account_id": 76281087,
"player_slot": 2,
"hero_id": 5,
"kills": 3,
"deaths": 13,
"assists": 10,
"last_hits": 22,
"denies": 0,
"gold_per_min": 215,
"xp_per_min": 259,
"level": 14,
},
{
"account_id": 4294967295,
"player_slot": 3,
"hero_id": 28,
"kills": 11,
"deaths": 11,
"assists": 11,
"last_hits": 166,
"denies": 18,
"gold_per_min": 413,
"xp_per_min": 485,
"level": 20,
},
{
"account_id": 81692493,
"player_slot": 4,
"hero_id": 2,
"kills": 1,
"deaths": 9,
"assists": 7,
"last_hits": 135,
"denies": 8,
"gold_per_min": 261,
"xp_per_min": 314,
"level": 16,
},
{
"account_id": 10101141,
"player_slot": 128,
"hero_id": 30,
"kills": 7,
"deaths": 8,
"assists": 25,
"last_hits": 90,
"denies": 2,
"gold_per_min": 382,
"xp_per_min": 421,
"level": 18,
},
{
"account_id": 62101519,
"player_slot": 129,
"hero_id": 7,
"kills": 6,
"deaths": 8,
"assists": 20,
"last_hits": 305,
"denies": 0,
"gold_per_min": 556,
"xp_per_min": 585,
"level": 22,
},
{
"account_id": 134700328,
"player_slot": 130,
"hero_id": 4,
"kills": 17,
"deaths": 2,
"assists": 13,
"last_hits": 335,
"denies": 16,
"gold_per_min": 729,
"xp_per_min": 724,
"level": 25,
},
{
"account_id": 35357393,
"player_slot": 131,
"hero_id": 83,
"kills": 4,
"deaths": 4,
"assists": 23,
"last_hits": 16,
"denies": 4,
"gold_per_min": 318,
"xp_per_min": 407,
"level": 18,
},
{
"account_id": 4294967295,
"player_slot": 132,
"hero_id": 101,
"kills": 13,
"deaths": 8,
"assists": 12,
"last_hits": 57,
"denies": 3,
"gold_per_min": 390,
"xp_per_min": 405,
"level": 18,
}
]
,
"radiant_win": false,
"duration": 2682,
"start_time": 1461781997,
"match_id": 2324299045,
"match_seq_num": 2036251155,
"cluster": 133,
"game_mode": 1,
"flags": 0,
"engine": 1,
"radiant_score": 30,
"dire_score": 48
}
Using an intelliJ plugin I have created 3 Java classes, one with the match result, one for the details of the result, and one for the details of the players within the result, each with the variables gets sets in:
TestMatch fields:
private TestMatchResult result;
TestMatchResult fields:
private int duration;
private int start_time;
private int cluster;
private boolean radiant_win;
private int match_seq_num;
private int engine;
private TestMatchResultPlayers[] players;
private long match_id;
private int dire_score;
private int flags;
private int game_mode;
private int radiant_score;
TestMatchResultPlayers fields:
private int kills;
private int gold_per_min;
private int last_hits;
private int account_id;
private int assists;
private int level;
private int player_slot;
private int xp_per_min;
private int hero_id;
private int denies;
private int deaths;
I have downloaded and added the gson library as a dependency into the intelliJ project.
I am trying to parse the json into the java classes as an object and would like to do that for all the match jsons, however I am not quite sure how to do that at the moment, all I have is:
public static void getMatch()
{
Gson gson = new Gson();
}
Could someone who understands gson better than myself give me a little bit of guidance as to how I'd go about parsing that json into the class(es) as an object for several match jsons? Once I've done that the rest of what I need to do is easy since it's just a case of taking the variables and running calculations on them then displaying them. If it's not possible or practical I can make a test CSV and read from that instead as I know how to use them, but only just come across jsons as that is what the valve API returns requests in so figured I may as well learn how to use them.
Thanks!

you need to use the method Gson.fromJson()
Example:
public static void getMatch()
{
Gson gson = new Gson();
TestMatch tm = gson.fromJson(jsonString, TestMatch.class);
}

Trouble deserializing Json from Google Analytics

Hi I'm attempting to deserializer.deserialize this data from Google Analytics
[[/s417, 14945, 93.17823577906019], [/s413, 5996, 72.57178438000356],
[/s417/, 3157, 25.690567351200837], [/s420, 2985, 44.12472727272727],
[/s418, 2540, 64.60275150472916], [/s416, 2504, 69.72643979057591],
[/s415, 2379, 44.69660861594867], [/s422, 2164, 57.33786505538772],
[/s421, 2053, 48.18852894317578], [/s414, 1839, 93.22588376273218],
[/s412, 1731, 54.8431860609832], [/s411, 1462, 71.26186830015314],
[/s419, 1423, 51.88551401869159], [/, 63, 11.303571428571429],
[/s420/, 22, 0.3333333333333333], [/s413/, 21, 7.947368421052632],
[/s416/, 16, 96.0], [/s421/, 15, 0.06666666666666667], [/s411/, 13,
111.66666666666667], [/s422/, 13, 0.07692307692307693], [/g150, 11, 0.09090909090909091], [/s414/, 10, 2.0], [/s418/, 10, 0.4444444444444444], [/s415/, 9, 0.2222222222222222], [/s412/, 8, 0.6666666666666666], [/s45, 6, 81.0], [/s164, 5, 45.25], [/s28, 5, 16.2], [/s39, 5, 25.2], [/s27, 4, 59.5], [/s29, 4, 26.5], [/s365, 3, 31.666666666666668], [/s506, 3, 23.333333333333332], [/s1139, 2, 30.5], [/s296, 2, 11.0], [/s311, 2, 13.5], [/s35, 2, 55.0], [/s363, 2, 15.5], [/s364, 2, 17.5], [/s419/, 2, 0.0], [/s44, 2, 85.5], [/s482, 2, 28.5], [/s49, 2, 29.5], [/s9, 2, 77.0], [/s146, 1, 13.0], [/s228, 1, 223.0], [/s229, 1, 54.0], [/s231, 1, 0.0], [/s30, 1, 83.0], [/s312, 1, 15.0], [/s313, 1, 155.0], [/s316, 1, 14.0], [/s340, 1, 22.0], [/s350, 1, 0.0], [/s362, 1, 24.0], [/s43, 1, 54.0], [/s442, 1, 87.0], [/s465,
1, 14.0], [/s468, 1, 67.0], [/s47, 1, 41.0], [/s71, 1, 16.0], [/s72,
1, 16.0], [/s87, 1, 48.0], [/s147, 0, 0.0], [/s417, 0, 0.0]]
With this
#Immutable
private static JSONDeserializer<List<List<String>>> deserializer = new JSONDeserializer<List<List<String>>>();
And it's failing silently on the deserialization.
Only error I'm getting is from the xhtml
com.sun.faces.context.PartialViewContextImpl$PhaseAwareVisitCallback
visit
SEVERE: javax.el.ELException: /views/guide/edit.xhtml #257,102 value="#{GuideEditController.visitsByScene}": flexjson.JSONException:
Missing value at character 2
Any clues?

marekful had the right idea
replaceAll("[^\d,[]\,]+", "") to remove the offending characters did the trick

How to resolve error "java.net.ConnectException: Connection refused: connect" in Storm Application while running in local cluster?

I am getting folowing error while running "real time feed processing with storm" application on local cluster. Link : http://www.datasalt.com/2012/01/real-time-feed-processing-with-storm/
4013 [main] INFO backtype.storm.zookeeper - Starting inprocess zookeeper at port 2000 and dir C:\Users\myname\AppData\Local\Temp\/94df123c-dd99-473b-b98e-66fbf59dd37c
4828 [main] INFO backtype.storm.daemon.nimbus - Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/a2319971-49a1-4767-a387-6ca302abc5de", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
4843 [main] INFO backtype.storm.daemon.nimbus - Using default scheduler
4999 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
5420 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
5655 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
5970 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6032 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
6110 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6110 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6235 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
6329 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6391 [main] INFO backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/d4fea730-5c16-4e20-a444-3769ee829406", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1 2 3), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
6454 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6517 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
6611 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6798 [main] INFO backtype.storm.daemon.supervisor - Starting supervisor with id 744a08ba-16c2-4043-81d0-a2fc45071fe1 at host DS-7071BC86F1CB.HCLT.CORP.HCL.IN
6798 [main] INFO backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/1116e43d-d550-484a-bf2e-d41968c48434", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (4 5 6), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
6813 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
6923 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
6985 [main] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting
7172 [main] INFO backtype.storm.daemon.supervisor - Starting supervisor with id 71fe3715-ba8e-40f2-a64b-cc00e46370bc at host DS-7071BC86F1CB.HCLT.CORP.HCL.IN
7266 [main] INFO backtype.storm.daemon.nimbus - Received topology submission for feedcount with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "topology.name" "feedcount", "storm.id" "feedcount-1-1410328057", "topology.workers" 2, "topology.debug" true}
7364 [main] INFO backtype.storm.daemon.nimbus - Activating feedcount: feedcount-1-1410328057
7588 [main] INFO backtype.storm.scheduler.EvenScheduler - Available slots: (["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 4] ["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 5] ["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 6] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 1] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 2] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 3])
7651 [main] INFO backtype.storm.daemon.nimbus - Setting new assignment for topology id feedcount-1-1410328057: #backtype.storm.daemon.common.Assignment{:master-code-dir "C:\\Users\\myname\\AppData\\Local\\Temp\\/a2319971-49a1-4767-a387-6ca302abc5de/nimbus/stormdist/feedcount-1-1410328057", :node->host {"744a08ba-16c2-4043-81d0-a2fc45071fe1" "DS-7071BC86F1CB.HCLT.CORP.HCL.IN", "71fe3715-ba8e-40f2-a64b-cc00e46370bc" "DS-7071BC86F1CB.HCLT.CORP.HCL.IN"}, :executor->node+port {[3 3] ["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 4], [6 6] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 1], [5 5] ["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 4], [4 4] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 1], [2 2] ["744a08ba-16c2-4043-81d0-a2fc45071fe1" 1], [1 1] ["71fe3715-ba8e-40f2-a64b-cc00e46370bc" 4]}, :executor->start-time-secs {[1 1] 1410328058, [5 5] 1410328058, [3 3] 1410328058, [2 2] 1410328058, [4 4] 1410328058, [6 6] 1410328058}}
7760 [Thread-5] INFO backtype.storm.daemon.supervisor - Downloading code for storm id feedcount-1-1410328057 from C:\Users\myname\AppData\Local\Temp\/a2319971-49a1-4767-a387-6ca302abc5de/nimbus/stormdist/feedcount-1-1410328057
7981 [Thread-8] INFO backtype.storm.daemon.supervisor - Downloading code for storm id feedcount-1-1410328057 from C:\Users\myname\AppData\Local\Temp\/a2319971-49a1-4767-a387-6ca302abc5de/nimbus/stormdist/feedcount-1-1410328057
8149 [Thread-5] INFO backtype.storm.daemon.supervisor - Finished downloading code for storm id feedcount-1-1410328057 from C:\Users\myname\AppData\Local\Temp\/a2319971-49a1-4767-a387-6ca302abc5de/nimbus/stormdist/feedcount-1-1410328057
8232 [Thread-6] INFO backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "feedcount-1-1410328057", :executors ([6 6] [4 4] [2 2])} for this supervisor 744a08ba-16c2-4043-81d0-a2fc45071fe1 on port 1 with id 66168bde-c945-46b3-8276-76598a2b7d3a
8234 [Thread-6] INFO backtype.storm.daemon.worker - Launching worker for feedcount-1-1410328057 on 744a08ba-16c2-4043-81d0-a2fc45071fe1:1 with id 66168bde-c945-46b3-8276-76598a2b7d3a and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/d4fea730-5c16-4e20-a444-3769ee829406", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm",
................
9689 [Thread-6] INFO backtype.storm.daemon.worker - Launching receive-thread for 744a08ba-16c2-4043-81d0-a2fc45071fe1:1
9704 [Thread-6] INFO backtype.storm.daemon.worker - Worker has topology config {"storm.id" "feedcount-1-1410328057", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/d4fea730-5c16-4e20-a444-3769ee829406", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "feedcount", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1 2 3), "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
9718 [Thread-6] INFO backtype.storm.daemon.worker - Worker 66168bde-c945-46b3-8276-76598a2b7d3a for storm feedcount-1-1410328057 on 744a08ba-16c2-4043-81d0-a2fc45071fe1:1 has finished loading
9711 [Thread-34-__system] INFO backtype.storm.daemon.executor - Preparing bolt __system:(-1)
9710 [Thread-32] INFO org.mortbay.log - Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
9757 [Thread-32] INFO org.mortbay.log - Reporting start.
9757 [Thread-32] INFO org.mortbay.log - Reporting finished.
9752 [Thread-34-__system] INFO backtype.storm.daemon.executor - Prepared bolt __system:(-1)
9763 [Thread-9] INFO backtype.storm.daemon.executor - Loading executor __acker:[1 1]
9764 [Thread-9] INFO backtype.storm.daemon.task - Emitting: __acker __system ["startup"]
9765 [Thread-9] INFO backtype.storm.daemon.executor - Loaded executor tasks __acker:[1 1]
9766 [Thread-9] INFO backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[1 1]
9767 [Thread-9] INFO backtype.storm.daemon.executor - Finished loading executor __acker:[1 1]
9767 [Thread-9] INFO backtype.storm.daemon.worker - Launching receive-thread for 71fe3715-ba8e-40f2-a64b-cc00e46370bc:4
9772 [Thread-9] INFO backtype.storm.daemon.worker - Worker has topology config {"storm.id" "feedcount-1-1410328057", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\myname\\AppData\\Local\\Temp\\/1116e43d-d550-484a-bf2e-d41968c48434", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "feedcount", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (4 5 6), "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
9772 [Thread-9] INFO backtype.storm.daemon.worker - Worker d01f3c23-f1c9-4fad-b5b2-540e932f98a8 for storm feedcount-1-1410328057 on 71fe3715-ba8e-40f2-a64b-cc00e46370bc:4 has finished loading
9801 [Thread-38-__acker] INFO backtype.storm.daemon.executor - Preparing bolt __acker:(1)
9801 [Thread-38-__acker] INFO backtype.storm.daemon.executor - Prepared bolt __acker:(1)
9805 [Thread-21-__acker] INFO backtype.storm.daemon.executor - Processing received message source: feedSpout:3, stream: __ack_init, id: {}, [1491166814217847811, -3901990751820783443, 3]
10757 [Thread-32] INFO org.mortbay.log - Reporting start.
10776 [Thread-32] INFO org.mortbay.log - Reporting finished.
**java.net.ConnectException: Connection refused: connect**
11065 [Thread-27-fetcherBolt] INFO backtype.storm.daemon.task - Emitting: fetcherBolt __ack_fail [1491166814217847811]
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at org.rometools.fetcher.impl.HttpURLFeedFetcher.retrieveFeed(HttpURLFeedFetcher.java:150)
at org.rometools.fetcher.impl.HttpURLFeedFetcher.retrieveFeed(HttpURLFeedFetcher.java:87)
at datasalt.storm.feeds.FetcherBolt.execute(FetcherBolt.java:62)
at backtype.storm.daemon.executor$fn__3498$tuple_action_fn__3500.invoke(executor.clj:615)
at backtype.storm.daemon.executor$mk_task_receiver$fn__3421.invoke(executor.clj:383)
at backtype.storm.disruptor$clojure_handler$reify__2962.onEvent(disruptor.clj:43)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:82)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:61)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at backtype.storm.daemon.executor$fn__3498$fn__3510$fn__3557.invoke(executor.clj:730)
at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
11068 [Thread-21-__acker] INFO backtype.storm.daemon.executor - Processing received message source: fetcherBolt:5, stream: __ack_fail, id: {}, [1491166814217847811]
11068 [Thread-21-__acker] INFO backtype.storm.daemon.task - Emitting direct: 3; __acker __ack_fail [1491166814217847811]

Criteria , have possibilities?

Well, I have the following questions, to perform the join between the tables by setting the nickname (alias), I need to make a decode, used the alias alias, but to use because it does not recognize the use of pure sql.
How do I return the name that defines the criteria for the tables? I'm using sqlGroupProjection, if you can suggest another way.
Criteria criteria = dao.getSessao().createCriteria(Chamado.class,"c");
criteria.createAlias("c.tramites","t").setFetchMode("t", FetchMode.JOIN);
projetos.add( Projections.rowCount(),"qtd");
criteria.add(Restrictions.between("t.dataAbertura", Formata.getDataD(dataInicio, "dd/MM/yyyy"), Formata.getDataD(dataFim, "dd/MM/yyyy")));
projetos.add(Projections.sqlGroupProjection("decode(t.cod_estado, 0, 0, 1, 1, 2, 1, 3, 2, 4, 1, 5, 3) as COD_ESTADO",
"decode(t.cod_estado, 0, 0, 1, 1, 2, 1, 3, 2, 4, 1, 5, 3)",
new String[]{"COD_ESTADO"},
new Type[]{Hibernate.INTEGER}));
criteria.setProjection(projetos);
List<Relatorio> relatorios = criteria.setResultTransformer(Transformers.aliasToBean(Relatorio.class)).list();
SQL generated by criteria:
select count(*) as y0_,
decode(t.cod_estado, 0, 0, 1, 1, 2, 1, 3, 2, 4, 1, 5, 3) as COD_ESTADO
from CHAMADOS this_
inner join TRAMITES t1_ on this_.COD_CHAMADO = t1_.COD_CHAMADO
where t1_.DT_ABERTURA between ? and ?
group by decode(t.cod_estado, 0, 0, 1, 1, 2, 1, 3, 2, 4, 1, 5, 3)

I get different results reading the same file from the file system and from inside a jar

I have a file that my Java application takes as input which I read 6 bytes at a time. When I read it in off the file system everything works fine. If I build everything into a jar the first 4868 reads work fine, but after that it starts returning the byte arrays in the wrong order and also ends up having read more data at the end.
Here is a simplified version of my code which reproduces the problem:
InputStream inputStream = this.getClass().getResourceAsStream(filePath);
byte[] byteArray = new byte[6];
int counter = 0;
while ((inputStream.read(byteArray) != -1))
{
counter++;
System.out.println("Read #" + counter +": " + Arrays.toString(byteArray));
}
System.out.println("Done.");
This is the [abbreviated] output I get when reading off of the file system:
...
Read #4867: [5, 0, 57, 7, 113, -26]
Read #4868: [2, 0, 62, 7, 114, -26]
Read #4869: [2, 0, 68, 7, 115, -26]
Read #4870: [3, 0, 75, 7, 116, -26]
Read #4871: [2, 0, 83, 7, 117, -26]
...
Read #219687: [1, 0, 4, -8, 67, 33]
Read #219688: [1, 0, 2, -8, 68, 33]
Read #219689: [5, 0, 1, -8, 67, 33]
Done.
And here is what I get reading from a jar:
...
Read #4867: [5, 0, 57, 7, 113, -26]
Read #4868: [2, 0, 62, 7, 113, -26] //everything is fine up to this point
Read #4869: [7, 114, -26, 2, 0, 68]
Read #4870: [7, 115, -26, 3, 0, 75]
Read #4871: [7, 116, -26, 2, 0, 83]
...
Read #219687: [95, 33, 1, 0, 78, -8]
Read #219688: [94, 33, 1, 0, 76, -8]
Read #219689: [95, 33, 1, 0, 74, -8]
...
Read #219723: [67, 33, 1, 0, 2, -8]
Read #219724: [68, 33, 5, 0, 1, -8]
Read #219725: [67, 33, 5, 0, 1, -8]
Done.
I unzipped the jar and confirmed that the files being read are identical, so what could cause the reader to return different results?

Your reading loop is wrong.
inputStream.read() method returns number of bytes it really read. You have to check this number before transforming the data into string.
When you are reading from file the bytes are not arrived all together. At one of the iterations of your loop you probably read 4 of expected 6 bytes, so your transformation to string does not work.
If you are reading integers I'd recommend you to wrap your raw input string using Scanner or good old DataInputStream and read integers directly.

Categories