We Keep Coding

How to convert string to MarketDataIncrementalRefresh in QUICKFIX in Java?

I am working on the Stock and Exchange Markets. I have a situation like : I need to take a string from the log and convert it to "Message" type Object. As per this link I have tried using all the three methods of the "MessageUtils" class in JAVA. But my String is being stripped to a Message class type object with unique tags. But as my string is "MarketDataIncrementalRefresh" type I want each every tag to be present in the Message.
For example : I am providing the following string to "MessageUtils.parse()" method.
8=FIX.4.2|9=00795|35=W|49=TT_PRICE|56=SAP0094X|34=2392|52=20170623-04:41:33.375|55=CL|48=00A0HR00CLZ|10455=CLQ7|167=FUT|207=CME|15=USD|262=MDRQ-751|200=201708|18210=1|387=12292|268=24|269=0|290=1|270=4290|271=33|269=0|290=2|270=4289|271=34|269=0|290=3|270=4288|271=40|269=0|290=4|270=4287|271=38|269=0|290=5|270=4286|271=46|269=0|290=6|270=4285|271=53|269=0|290=7|270=4284|271=46|269=0|290=8|270=4283|271=66|269=0|290=9|270=4282|271=48|269=0|290=10|270=4281|271=64|269=1|290=1|270=4291|271=21|269=1|290=2|270=4292|271=40|269=1|290=3|270=4293|271=48|269=1|290=4|270=4294|271=83|269=1|290=5|270=4295|271=62|269=1|290=6|270=4296|271=46|269=1|290=7|270=4297|271=34|269=1|290=8|270=4298|271=55|269=1|290=9|270=4299|271=31|269=1|290=10|270=4300|271=128|269=2|270=4291|271=1|269=4|270=4280|269=7|270=4292|269=8|270=4277|10=044|
But what I am getting is this:
8=FIX.4.2|9=192|35=W|34=2|49=TT_PRICE|52=20170622-14:16:23.685|56=SAP0094X|15=USD|48=00A0HR00GCZ|55=GC|167=FUT|200=201708|207=CME|262=MDRQ-21|268=25|269=0|270=12510|271=24|290=1|387=121890|10455=GCQ7|18210=1|10=036|
As you can observe only unique tags are present in the String. But I want each and every tag , no matter how many times it exists in the provided string.
Please can anyone help me doing this in JAVA. It will be really appreciable.
Below is the code I am using for converting :
MessageUtils mu = new MessageUtils();
Session session = Session.lookupSession(sessionID);
Message msg = MessageUtils.parse(new DefaultMessageFactory(), null, str);
// Message msg = new Message(str, false); //This can also be used for converting
System.out.println(msg.toString());

The other thread says:
MessageUtils.parse(MessageFactory messageFactory, DataDictionary dataDictionary, java.lang.String messageString)
And your code says:
Message msg = MessageUtils.parse(new DefaultMessageFactory(), null, str);
So you need to fix your data dictionary and pass it to the parse method instead of passing 'null'

I think the problem is as follows. There's a repeating group that starts with tag 286 (NoMDEntries). The order of fields in a repeating group should be strict, i.e. the same order as the definition of the repeating group. See Market Data - Snapshot/Full Refresh or the data dictionnary supplied by QuickFIX/J (FIX42.xml).
The 268 tag should be followed by 269 and then 270. I am seeing in your message string: |268=24|269=0|290=1|270=4290| which is the incorrect order of tags. That is probably the reason why the message is truncated by MessageUtils.parse.
As a test you could try to manually correct the order in the string and try parsing that to see if that gives the correct message.

Elasticsearch SearchContextMissingException during 'scan & scroll' query with Spring Data Elasticsearch

I am using elasticsearch 2.2.0 with default cluster configuration. I encounter a problem with scan and scroll query using spring data elasticsearch. When I execute the query I get error like this:
[2016-06-29 12:45:52,046][DEBUG][action.search.type ] [Vector] [155597] Failed to execute query phase
RemoteTransportException[[Vector][10.132.47.95:9300][indices:data/read/search[phase/scan/scroll]]]; nested: SearchContextMissingException[No search context found for id [155597]];
Caused by: SearchContextMissingException[No search context found for id [155597]]
at org.elasticsearch.search.SearchService.findContext(SearchService.java:611)
at org.elasticsearch.search.SearchService.executeScan(SearchService.java:311)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchScanScrollTransportHandler.messageReceived(SearchServiceTransportAction.java:433)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchScanScrollTransportHandler.messageReceived(SearchServiceTransportAction.java:430)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
My 'scan & scroll' code:
public List<T> getAllElements(SearchQuery searchQuery) {
searchQuery.setPageable(new PageRequest(0, PAGE_SIZE));
String scrollId = elasticsearchTemplate.scan(searchQuery, 1000, false);
List<T> allElements = new LinkedList<>();
boolean hasRecords = true;
while (hasRecords) {
Page<T> page = elasticsearchTemplate.scroll(scrollId, 5000, resultMapper);
if (page.hasContent()) {
allElements.addAll(page.getContent());
} else {
hasRecords = false;
}
}
elasticsearchTemplate.clearScroll(scrollId);
return allElements;
}
When my query result size is less than PAGE_SIZE parameter, then error like this occurs five times. I guess that it is one per shard. When result size is bigger than PAGE_SIZE, the error occurs few times more. I've tried to refactor my code to not call:
Page<T> page = elasticsearchTemplate.scroll(scrollId, 5000, resultMapper);
when I'm sure that the page has no content. But it works only if PAGE_SIZE is bigger than query result, so it is no the solution at all.
I have to add that it is problem only on elasticsearch side. On the client side the errors is hidden and in each case the query result is correct. Has anybody knows what causes this issue?
Thank you for help,
Simon.

I get this error if the ElasticSearch system closes the connection. Typically it's exactly what #Val said - dead connections. Things sometimes die in ES for no good reason - master node down, data node is too congested, bad performing queries, Kibana running at the same time you are in middle of querying...I've been hit by all of these at one time or another to get this error.
Suggestion: Up the initial connection time - 1000L might be too short for it to get what it needs. It won't hurt if the query ends sooner.
This also happens randomly when I try to pull too much data quickly; you might have huge documents and trying to pull PAGESIZE of 50,000 might be a little too much. We don't know what you chose for PAGESIZE.
Suggestion: Lower PAGESIZE to something like 500. Or 20. See if these smaller values slow down the errors.
I know I have less of these problems after moving to ES 2.3.3.

This usually happens if your search context is not alive anymore.
In your case, you're starting your scan with a timeout of 1 second and then each scan is alive during 5 seconds. It's probably too low. The default duration to keep the search context alive is 1 minute, so you should probably increase it to 60 seconds like this:
String scrollId = elasticsearchTemplate.scan(searchQuery, 60000, false);
...
Page<T> page = elasticsearchTemplate.scroll(scrollId, 60000, resultMapper);

I ran into a similar problem and I suspect that Spring Data Elasticsearch has some internal bug about passing the Scroll-ID. In my case I just tried to scroll through the whole index and I can rule out #Val his answer about "This usually happens if your search context is not alive anymore", because the exceptions occurred regardless of the duration. Also the exceptions started after the first Page and occurred for every other paging query.
In my case I could simply use elasticsearchTemplate.stream(). It uses Scroll & Scan internally and seems to pass the Scroll-ID correctly. Oh, and it's simpler to use:
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.matchAllQuery())
.withPageable(new PageRequest(0, 10000))
.build();
Iterator<Post> postIterator = elasticsearchTemplate.stream(searchQuery, Post.class);
while(postIterator.hasNext()) {
Post post = postIterator.next();
}

How LDAP search filter string accepts space?

I feel a bit nervous because this is my first question here at Stack Overflow. Please let me know if I am not doing it in a good manner.
In LDAP, I think the following search filter string works.
( & (uid=tt4cs) (objectClass=inetOrgPerson) )
It means searching for entries, one of whose uid is tt4cs and one of whose objectClass is inetOrgPerson.
Please note that there are spaces between every parenthesis and ampersand, which will just be ignored. But, as far as I read RFC4515, I can find no implication that allows any space that way. Could anybody kindly tell me whether it is allowed by any other standards or it is just so by convention?
Update on Jan 13, 2014
I have tested it in three ways. (LDAP server in my environment is OpenLDAP 2.4.38)
(1) Do ldapsearch on command line. The above search filter works and gets a result.
(2) Search by using UnboundID LDAP SDK for Java. This API does not send the search request to the server, but throws an exception that says "Unexpected closing parenthesis found at position 15 of the filter string."
String filter = "( & (uid=tt4cs) (objectClass=inetOrgPerson) )";
SearchResult searchResult
= connection.search("dc=localdomain", SearchScope.SUB, filter);
(3) Search by using Apache Directory LDAP API. This API does not send the search request to the server, but throws an exception that says "The filter ( & (uid=tt4cs) (objectClass=inetOrgPerson) ) is invalid."
String filter = "( & (uid=tt4cs) (objectClass=inetOrgPerson) )";
EntryCursor cursor
= connection.search("dc=localdomain", filter, SearchScope.SUBTREE);
Now I have a feeling that acceptance of the extra spaces may probably be an implementation-dependent behavior, and that it is better to avoid it.

Faceting using SolrJ and Solr4

I've gone through the related questions on this site but haven't found a relevant solution.
When querying my Solr4 index using an HTTP request of the form
&facet=true&facet.field=country
The response contains all the different countries along with counts per country.
How can I get this information using SolrJ?
I have tried the following but it only returns total counts across all countries, not per country:
solrQuery.setFacet(true);
solrQuery.addFacetField("country");
The following does seem to work, but I do not want to have to explicitly set all the groupings beforehand:
solrQuery.addFacetQuery("country:usa");
solrQuery.addFacetQuery("country:canada");
Secondly, I'm not sure how to extract the facet data from the QueryResponse object.
So two questions:
1) Using SolrJ how can I facet on a field and return the groupings without explicitly specifying the groups?
2) Using SolrJ how can I extract the facet data from the QueryResponse object?
Thanks.
Update:
I also tried something similar to Sergey's response (below).
List<FacetField> ffList = resp.getFacetFields();
log.info("size of ffList:" + ffList.size());
for(FacetField ff : ffList){
String ffname = ff.getName();
int ffcount = ff.getValueCount();
log.info("ffname:" + ffname + "|ffcount:" + ffcount);
}
The above code shows ffList with size=1 and the loop goes through 1 iteration. In the output ffname="country" and ffcount is the total number of rows that match the original query.
There is no per-country breakdown here.
I should mention that on the same solrQuery object I am also calling addField and addFilterQuery. Not sure if this impacts faceting:
solrQuery.addField("user-name");
solrQuery.addField("user-bio");
solrQuery.addField("country");
solrQuery.addFilterQuery("user-bio:" + "(Apple OR Google OR Facebook)");
Update 2:
I think I got it, again based on what Sergey said below. I extracted the List object using FacetField.getValues().
List<FacetField> fflist = resp.getFacetFields();
for(FacetField ff : fflist){
String ffname = ff.getName();
int ffcount = ff.getValueCount();
List<Count> counts = ff.getValues();
for(Count c : counts){
String facetLabel = c.getName();
long facetCount = c.getCount();
}
}
In the above code the label variable matches each facet group and count is the corresponding count for that grouping.

Actually you need only to set facet field and facet will be activated (check SolrJ source code):
solrQuery.addFacetField("country");
Where did you look for facet information? It must be in QueryResponse.getFacetFields (getValues.getCount)

In the solr Response you should use QueryResponse.getFacetFields() to get List of FacetFields among which figure "country". so "country" is idenditfied by QueryResponse.getFacetFields().get(0)
you iterate then over it to get List of Count objects using
QueryResponse.getFacetFields().get(0).getValues().get(i)
and get value name of facet using QueryResponse.getFacetFields().get(0).getValues().get(i).getName()
and the corresponding weight using
QueryResponse.getFacetFields().get(0).getValues().get(i).getCount()

Why wont this sort in Solr work?

I need to sort on a date-field type, which name is "mod_date".
It works like this in the browser adress-bar:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+desc
But I am using a phpSolr client which sends an URL to Solr, and the url sent is this:
fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc
// This wont work and is echoed after this in php:
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
This wont work, I dont know why!
Everything else works fine, all right fields are returned. But the sort doesn't work.
Any ideas?
Thanks
BTW: The field "mod_date" contains something like:
2010-03-04T19:37:22.5Z
EDIT:
First I use PHP to send this to a SolrPhpClient which is another php-file called service.php:
require_once('../SolrPhpClient/Apache/Solr/Service.php');
$solr = new Apache_Solr_Service('localhost', 8983, '/solr/');
$results = $solr->search($querystring, $p, $limit, $solr_params);
$solr_params is an array which contains the solr-parameters (q, fq, etc).
Now, in service.php:
$params['version'] = self::SOLR_VERSION;
// common parameters in this interface
$params['wt'] = self::SOLR_WRITER;
$params['json.nl'] = $this->_namedListTreatment;
$params['q'] = $query;
$params['sort'] = 'mod_date desc'; // HERE IS THE SORT I HAVE PROBLEM WITH
$params['start'] = $offset;
$params['rows'] = $limit;
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
if ($method == self::METHOD_GET)
{
return $this->_sendRawGet($this->_searchUrl . $this->_queryDelimiter . $queryString);
}
else if ($method == self::METHOD_POST)
{
return $this->_sendRawPost($this->_searchUrl, $queryString, FALSE, 'application/x-www-form-urlencoded');
}
The $results contain the results from Solr...
So this is the way I need to get to work (via php).
This code below (also on top of this Q) works but thats because I paste it into the adress bar manually, not via the PHPclient. But thats just for debugging, I need to get it to work via the PHPclient:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+des // Not via phpclient, but works
UPDATE (2010-03-08):
I have tried Donovans codes (the urls) and they work fine.
Now, I have noticed that it is one of the parameters causing the 'SORT' not to work.
This parameter is the "wt" parameter. If we take the url from top of this Q, (fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc), and just simply remove the "wt" parameter, then the sort works.
BUT the results appear differently, thus making my php code not able to recognize the results I believe. Donovan would know this I think. I am guessing in order for the PHPClient to work, the results must be in a specific structure, which gets messed up as soon as I remove the wt parameter.
Donovan, help me please...
Here is some background what I use your SolrPhpClient for:
I have a classifieds website, which uses MySql. But for the searching I am using Solr to search some indexed fields. Then Solr returns an array of ID:numbers (for all matches of the search criteria). Then I use those ID:numbers to find everything in a MySql db and fetch all other information (example is not searchable information).
So simplified: Search -> Solr returns all matches in an array of ID:nrs -> Id:numbers from Solr are the same as the Id numbers in the MySql db, so I can just make a simple match agains every record with the ID matching the ID from the Solr results array.
I don't use Faceting, no boosting, no relevancy or other fancy stuff. I only sort by the latest classified put, and give the option to users to also sort on the cheapest price. Nothing more.
Then I use the "fq" parameter to do queries on different fields in Solr depending on category chosen by users (example "cars" in this case which in my language is "Bilar").
I am really stuck with this problem here...
Thanks for all help

As pointed out in the stack overflow comments, your browser query is different than your php client based query - to remove that from the equation you should test with this corrected. To get the same results as the browser based query you're php code should have looked something like this:
$solr = new Apache_Solr_Client(...);
$searchOptions = array(
'sort' => 'mod_date desc'
);
$results = $solr->search("bmw", 0, 10, $searchOptions);
Instead, I imagine it looks more like:
$searchOptions = array(
'fq' => 'category:"Bilar" + car_action:Sälje',
'sort' => 'mod_date desc'
)
$solr->search("\*:*", 0, 10, $searchOptions);
What I expect you to see is that php client results will be the same as the browser based results, and I imagine the same would happen if you did it the opposite way - take your current parameters from the php client and applied them correctly to the browser based query.
Now onto your problem, you don't see documents sorted properly.
I would try this query, which is the equivalent of the php client based code:
http://localhost:8983/solr/select/?&q=%2A%3A%2A&fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
versus this query, which moves the filter query into the main query:
http://localhost:8983/solr/select/?&q=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
and see if there is a difference. If there is, then it might be a bug in how results from cached filtered queries are used and sorted by solr - which wouldn't be a problem with the client, but the solr service itself.
Hope this gets you closer to an anser.

Use session's values for save sort parameters.

The quick answer in case someone is attempting to sort via solr-php-client:
$searchOptions = array('sort' => 'field_date desc');
Ditch the + sign that you would usually put on the URL. It took me a while as well to figure it out, I was encoding it and putting it all over the place...

It's possible it's related to the json.nl=map parameter. When the response is set to JSON with wt=json and json.nl=map, facets are not sorted as expected with the facet.sort or f.<field_name>.facet.sort=count|index options.
e.g. with facet.sort=count and wt=json only, I get:
"dc_coverage": [
"United States",
5,
"19th century",
1,
"20th century",
1,
"Detroit (Mich.)",
1,
"Pennsylvania",
1,
"United States--Michigan--Detroit",
1,
"United States--Washington, D.C.",
1
]
But with facet.sort=count, wt=json, and json.nl=map as an option, you can see the sorting is lost:
"dc_coverage": {
"19th century": 1,
"20th century": 1,
"Detroit (Mich.)": 1,
"Pennsylvania": 1,
"United States": 5,
"United States--Michigan--Detroit": 1,
"United States--Washington, D.C.": 1
}
There is more information here about formatting the JSON response when using json.nl=map: https://cwiki.apache.org/confluence/display/solr/Response+Writers#ResponseWriters-JSONResponseWriter