I've done my best to find the answer, but nothings seems to cover what I'm trying to do. I know how to get a single feed, from say a fan page like Starbucks, or Pepsi using the Graph API. What I want to do is get several, or multiple, feeds in a single request. Is this possible? I was told by another developer that it is, but haven't found anything on this subject in the FB documentation, or anywhere else online. I have like 50 feeds I want to aggregate into a single feed, and I don't want to make 50 different requests to get all that data.
Has anyone know how to do this, or know if it's even possible?
Thanks.
After ever MORE searching, apparently you can do this. You do this by using Batch Requests:
https://developers.facebook.com/docs/reference/api/batch/
Hope this will save someone time in the future :)
Looks like you can't really.
This works:
SELECT page_id
FROM page
WHERE username='facebook'
OR username='coca-cola'
{
"data": [
{
"page_id": 20531316728
},
{
"page_id": 40796308305
}
]
}
This works (~100 results):
SELECT post_id,message
FROM stream
WHERE source_id IN (
SELECT page_id
FROM page
WHERE username='facebook'
) LIMIT 100
But this doesn't really works (3 results):
SELECT post_id,message
FROM stream
WHERE source_id IN (
SELECT page_id
FROM page
WHERE username='facebook'
OR username='coca-cola'
) LIMIT 100
Related
Currently I am using the gcs-text-to-bigquery google provided template and feeding in a transform function to transform my jsonl file. The jsonl is pretty nested and i wanted to be able to output multiple rows per one row of the newline delimited json by doing some transforms.
For example:
{'state': 'FL', 'metropolitan_counties':[{'name': 'miami dade', 'population':100000}, {'name': 'county2', 'population':100000}…], 'rural_counties':{'name': 'county1', 'population':100000}, {'name': 'county2', 'population':100000}….{}], 'total_state_pop':10000000,….}
There will obviously be more counties than 2 and each state will have one of these lines. The output my boss wants is:
When i do the gcs-to-bq text transform, i end up only getting one line per state (so I'll get miami dade county from FL, and then whatever the first county is in my transform for the next state). I read a little bit and i think this is because of the mapping in the template that expects one output per jsonline. It seems I can do a pardo(DoFn ?) not sure what that is, or there is a similar option with beam.Map in python. There is some business logic in the transforms (right now it's about 25 lines of code as the json has more columns than i showed but those are pretty simple).
Any suggestions on this? data is coming in tonight/tomorrow, and there will be hundreds of thousands of rows in a BQ table.
the template i am using is currently in java, but i can translate it to python pretty easily as there are a lot of examples online in python. i know python better and i think its easier given the different types (sometimes a field can be null) and it seems less daunting given the examples i saw look simpler, however, open to either
Solving that in Python is somewhat straightforward. Here's one possibility (not fully tested):
from __future__ import absolute_import
import ast
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/service_account.json'
pipeline_args = [
'--job_name=test'
]
pipeline_options = PipelineOptions(pipeline_args)
def jsonify(element):
return ast.literal_eval(element)
def unnest(element):
state = element.get('state')
state_pop = element.get('total_state_pop')
if state is None or state_pop is None:
return
for type_ in ['metropolitan_counties', 'rural_counties']:
for e in element.get(type_, []):
name = e.get('name')
pop = e.get('population')
county_type = (
'Metropolitan' if type_ == 'metropolitan_counties' else 'Rural'
)
if name is None or pop is None:
continue
yield {
'State': state,
'County_Type': county_type,
'County_Name': name,
'County_Pop': pop,
'State_Pop': state_pop
}
with beam.Pipeline(options=pipeline_options) as p:
lines = p | ReadFromText('gs://url to file')
schema = 'State:STRING,County_Type:STRING,County_Name:STRING,County_Pop:INTEGER,State_Pop:INTEGER'
data = (
lines
| 'Jsonify' >> beam.Map(jsonify)
| 'Unnest' >> beam.FlatMap(unnest)
| 'Write to BQ' >> beam.io.Write(beam.io.BigQuerySink(
'project_id:dataset_id.table_name', schema=schema,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
)
)
This will only succeed if you are working with batch data. If you have streaming data then just change beam.io.Write(beam.io.BigquerySink(...)) to beam.io.WriteToBigQuery.
Good time of day ,
I have a question about using jsoup.
Working with one web site I met problem that before gathering information from list of products at first I need to select radio buttons and send required data about two parameters of products : "type" and "categories".
This web site if so:
http://h17007.www1.hpe.com/us/en/networking/products/eos/index.aspx#.WzYf4NIzY2z
Because if not to do it search will give list which have only one product from unknown category .
This:
HP VCX V7205 Platform with DL 360 G6
I'm using such method at the begining.
Response response = Jsoup.connect("http://h17007.www1.hpe.com/us/en/networking/products/eos/index.aspx#.WzSdxdIzY2z")
.userAgent("Mozilla/5.0")
.timeout(10 * 1000)
.method(Method.POST)
.data("type","2")
.data("category","4")
.followRedirects(true)
.execute();
And then parse it.
Document document = response.parse();
But unfortunatly either I didn't understand how to send it correctly or I should choose another way of ,maybe , selecting correct list of products .As you may understand I try to choose second type and fouth category.
Could somebody advise how to do it ? Or advise example which could be useful for may case ?
Thank you for your attention
I'm trying to fill a combo using the helper #select of Play! in a form, but searching a lot, I not found what I exactly need.
I want something like this:
#select(
filmeForm("Director"),
options(Seq(aListOfDirectors))
)
This is a form of register of movies, that get Foreign key of a Director.
I need list the directors names, and when I send the form, I need to get the ID of selected director.
If this way is not possible, some similar way will be useful.
Can anyone help me?
Thanks in advance.
HTML select tag can have a set of option tags like the following:
<select id="directors-select" name="director">
<option value="steven-spilberg">Steven Spilberg</option>
<option value="stanley-kubric">Stanley Kubric</option>
</select>
So, to proper populate select's options, Play #select helper requires that the Seq contains a tuple (String, String), which will contains both the value attribute and also the "label" presented to the user. In other words, options parameter needs to be a Seq[(String, String)]. Here is the example given at the docs:
#select(
field = myForm("mySelect"),
options = Seq(
"Foo" -> "foo text",
"Bar" -> "bar text",
"Baz" -> "baz text"
),
'_default -> "Choose One",
'_disabled -> Seq("FooKey", "BazKey")
'cust_att_name -> "cust_att_value"
)
So, your aListOfDirectors needs to contains a (String, String) tuple. But it is actually pretty simple to solve this, just change your code to:
#select(
field = filmeForm("Director"),
options = aListOfDirectors.map(director => director.id.toString -> director.name)
)
Here, I'm considering that aListOfDirectors is a Seq[Director].
Where is it documented?
Play documentation for (Java) forms states that "there are several input helpers in the views.html.helper package." After that, I just looked up at the play scaladocs and then navigate to views.html.helper package. There you can find the docs for #select.
I know that when people says "read the docs" that sometimes sounds harsh, but it is a good advice considering that framework/software/lib developers (who really knows the framework/software/lib) spent their time to explain how to use the framework/software/lib, we for sure can spend some time reading the docs.
I have developed code as below.
jQuery(document).ready(function(){
jQuery("#records").jqGrid({
height:350,
datatype: 'local',
colNames:['Policy Name','Policy Type', 'Time allowed (HH:mm)','Expiration Duration (days)','Session Pulse(minutes)','Description'],
colModel :[
{name:'pName', index:'pName', editable:true,sorttype:'text',width:150,editoptions:{size:10},formatter:'showlink',formatoptions:{baseLinkUrl:'javascript:' , showAction: "GetAndShowUserData(jQuery('#list2'),'",addParam: "');"}},
{name:'pType', index:'pType', sorttype:'text',editable:true,width:150,editoptions:{size:10}},
{name:'timeAllowed', index:'timeAllowed', sorttype:'text',editable:true,width:200, align:"right",editoptions:{size:10}},
{name:'expDuration', index:'expDuration', sorttype:'text',editable:true,width:200, align:"right",editoptions:{size:10}},
{name:'sessionPulse', index:'sessionPulse',sorttype:'int',editable:true,width:200, align:"right",editoptions:{size:10}},
{name:'description', index:'description', sortable:false,editable:true,width:200,editoptions:{size:10}}],
pager:jQuery('#pager'),
rowNum:10,
sortname: 'pName',
autowidth:true,
altRows:true,
drag:true,
sortorder: "asc",
rowList:[2,5,10,20],
viewrecords: true,
loadonce:false,
multiselect: true,
/*
onSelectRow: function(id){
var gr = jQuery("#records").jqGrid('getGridParam','selrow');
if( gr != null ) jQuery("#records").jqGrid('editGridRow',gr,{height:280,reloadAfterSubmit:false});
else alert("Please Select Row");
},
editurl: "server.php",
*/
caption:'Manage Policy'
});
});
Now, I want to make an Ajax request to the servlet for next records when the user presses >> the (next) button of the jqGrid. I have searched much on the Internet, but I found a lot of PHP code, but I can not understand that PHP; I want to develop that thing in Java. How can I do that?
As GPS said, the paging in jqGrid works by paging through its current dataset. You have to load a big set of data, and it will page through that dataset. There may be a way to get it to behave the way you want, but I don't know how.
For my grids, I use the pagination plugin to trigger an Ajax call to get the next page of data. When the data comes back, I just clear the grid (clearGridData) and add the new set of data using addRowData.
I'm a .NET programmer, so I don't know how you'd do the database calls in with Java, but that's not really a jqGrid question.
To determine how many pages there are, you take the count of all the records that you'll be paging through and divide that by the number of records you will show on the grid per page.
I need to sort on a date-field type, which name is "mod_date".
It works like this in the browser adress-bar:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+desc
But I am using a phpSolr client which sends an URL to Solr, and the url sent is this:
fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc
// This wont work and is echoed after this in php:
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
This wont work, I dont know why!
Everything else works fine, all right fields are returned. But the sort doesn't work.
Any ideas?
Thanks
BTW: The field "mod_date" contains something like:
2010-03-04T19:37:22.5Z
EDIT:
First I use PHP to send this to a SolrPhpClient which is another php-file called service.php:
require_once('../SolrPhpClient/Apache/Solr/Service.php');
$solr = new Apache_Solr_Service('localhost', 8983, '/solr/');
$results = $solr->search($querystring, $p, $limit, $solr_params);
$solr_params is an array which contains the solr-parameters (q, fq, etc).
Now, in service.php:
$params['version'] = self::SOLR_VERSION;
// common parameters in this interface
$params['wt'] = self::SOLR_WRITER;
$params['json.nl'] = $this->_namedListTreatment;
$params['q'] = $query;
$params['sort'] = 'mod_date desc'; // HERE IS THE SORT I HAVE PROBLEM WITH
$params['start'] = $offset;
$params['rows'] = $limit;
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
if ($method == self::METHOD_GET)
{
return $this->_sendRawGet($this->_searchUrl . $this->_queryDelimiter . $queryString);
}
else if ($method == self::METHOD_POST)
{
return $this->_sendRawPost($this->_searchUrl, $queryString, FALSE, 'application/x-www-form-urlencoded');
}
The $results contain the results from Solr...
So this is the way I need to get to work (via php).
This code below (also on top of this Q) works but thats because I paste it into the adress bar manually, not via the PHPclient. But thats just for debugging, I need to get it to work via the PHPclient:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+des // Not via phpclient, but works
UPDATE (2010-03-08):
I have tried Donovans codes (the urls) and they work fine.
Now, I have noticed that it is one of the parameters causing the 'SORT' not to work.
This parameter is the "wt" parameter. If we take the url from top of this Q, (fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc), and just simply remove the "wt" parameter, then the sort works.
BUT the results appear differently, thus making my php code not able to recognize the results I believe. Donovan would know this I think. I am guessing in order for the PHPClient to work, the results must be in a specific structure, which gets messed up as soon as I remove the wt parameter.
Donovan, help me please...
Here is some background what I use your SolrPhpClient for:
I have a classifieds website, which uses MySql. But for the searching I am using Solr to search some indexed fields. Then Solr returns an array of ID:numbers (for all matches of the search criteria). Then I use those ID:numbers to find everything in a MySql db and fetch all other information (example is not searchable information).
So simplified: Search -> Solr returns all matches in an array of ID:nrs -> Id:numbers from Solr are the same as the Id numbers in the MySql db, so I can just make a simple match agains every record with the ID matching the ID from the Solr results array.
I don't use Faceting, no boosting, no relevancy or other fancy stuff. I only sort by the latest classified put, and give the option to users to also sort on the cheapest price. Nothing more.
Then I use the "fq" parameter to do queries on different fields in Solr depending on category chosen by users (example "cars" in this case which in my language is "Bilar").
I am really stuck with this problem here...
Thanks for all help
As pointed out in the stack overflow comments, your browser query is different than your php client based query - to remove that from the equation you should test with this corrected. To get the same results as the browser based query you're php code should have looked something like this:
$solr = new Apache_Solr_Client(...);
$searchOptions = array(
'sort' => 'mod_date desc'
);
$results = $solr->search("bmw", 0, 10, $searchOptions);
Instead, I imagine it looks more like:
$searchOptions = array(
'fq' => 'category:"Bilar" + car_action:Sälje',
'sort' => 'mod_date desc'
)
$solr->search("\*:*", 0, 10, $searchOptions);
What I expect you to see is that php client results will be the same as the browser based results, and I imagine the same would happen if you did it the opposite way - take your current parameters from the php client and applied them correctly to the browser based query.
Now onto your problem, you don't see documents sorted properly.
I would try this query, which is the equivalent of the php client based code:
http://localhost:8983/solr/select/?&q=%2A%3A%2A&fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
versus this query, which moves the filter query into the main query:
http://localhost:8983/solr/select/?&q=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
and see if there is a difference. If there is, then it might be a bug in how results from cached filtered queries are used and sorted by solr - which wouldn't be a problem with the client, but the solr service itself.
Hope this gets you closer to an anser.
Use session's values for save sort parameters.
The quick answer in case someone is attempting to sort via solr-php-client:
$searchOptions = array('sort' => 'field_date desc');
Ditch the + sign that you would usually put on the URL. It took me a while as well to figure it out, I was encoding it and putting it all over the place...
It's possible it's related to the json.nl=map parameter. When the response is set to JSON with wt=json and json.nl=map, facets are not sorted as expected with the facet.sort or f.<field_name>.facet.sort=count|index options.
e.g. with facet.sort=count and wt=json only, I get:
"dc_coverage": [
"United States",
5,
"19th century",
1,
"20th century",
1,
"Detroit (Mich.)",
1,
"Pennsylvania",
1,
"United States--Michigan--Detroit",
1,
"United States--Washington, D.C.",
1
]
But with facet.sort=count, wt=json, and json.nl=map as an option, you can see the sorting is lost:
"dc_coverage": {
"19th century": 1,
"20th century": 1,
"Detroit (Mich.)": 1,
"Pennsylvania": 1,
"United States": 5,
"United States--Michigan--Detroit": 1,
"United States--Washington, D.C.": 1
}
There is more information here about formatting the JSON response when using json.nl=map: https://cwiki.apache.org/confluence/display/solr/Response+Writers#ResponseWriters-JSONResponseWriter