I have scenario where i need to pass customized parameters to solr data import query.
Ex- select * from customer where last_updated_date >=last_updated_indexed_date
The last_updated_indexed_date is coming from another table which has details about core.
How can I pass that last_indexed_updated_date in DIH query.
The data-config can be configured something like below :
<dataConfig>
<dataSource name="ds-db" driver="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:#127.0.0.1:1521:test" user="dev" password="dev" />
<dataSource name="ds-file" type="BinFileDataSource" />
<document name="documents">
<entity name="book" dataSource="ds-db"
query="select distinct
book.id as id,
book.title,
book.author,
book.publisher,
from Books book
where book.book_added_date >= to_date($ {dataimporter.request.lastIndexDate}, 'DD/MM/YYYY HH24:MI:SS')))"
transformer="DateFormatTransformer">
<field column=”id” name=”id” />
<field column=”title” name=”title” />
<field column=”author” name=”author” />
<field column=”publisher” name=”publisher” />
<entity name=”content” query=”select description from content
where content_id='${book.Id}' ”>
<field column=”description” name=”description” />
</entity>
</entity>
</document>
</dataConfig>
The way here '${book.Id}' is retrieved and passed to another query. You will also need to work upon something similar for the last_indexed_updated_date in your data-config.xml. if you don't have the same in your tables. You can try the same passing to the data import url like lastIndexDate(Please refer the below data import url.)
The data import url will be be like
http://localhost:8080/solr/admin/select/?qt=/dataimport&command=full-import&clean=false&commit=true&lastIndexDate='08/05/2011 20:16:11'
Related
I have this issue when querying certain objects (persisting in Postgres database)
using Castor 0.9.6 (can't upgrade)
Problem only occurs on production, not dev machine. Despite using same database.
udOql= db.getOQLQuery( "SELECT DISTINCT ud FROM model.objects.UserDefinition ud WHERE ud.siteId = $1");
udOql.bind(other.getSiteId());
This normally is working fine but for some reason I am getting duplicate objects, even when I include DISTINCT
Mapping XML is as follows:
<class name="model.objects.UserDefinition" identity="userId" key-generator="HIGH-LOW">
<cache-type type="unlimited" capacity="1000"/>
<map-to table="userdefinition" xml="userDefinition" />
<field name="siteId" type="java.lang.Integer">
<sql name="siteid" type="integer" />
</field>
<field name="userAttributeDefinition" type="model.objects.UserAttributeDefinition" collection="vector">
<sql many-key="userid" />
</field>
<field name="roleAssignmentDefinition" type="model.objects.RoleAssignmentDefinition" collection="vector">
<sql many-key="userid" />
</field>
</class>
Not only will it return multiple UserDefinition objects, some of them are missing some linked roleAssignmentDefinition or userAttributeDefinition. Usually after restarting tomcat it fixes the problem but then it comes back. Sometimes the issue the issue is there from the start.
The database holds correct data. Any tips, pointers much appreciated
I have developed Odata service for a system entity which generates a metadata but however I cant figure out how to add Annotations element to it. Sample Metadata generated is as follows :-
<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx" xmlns:sap="http://www.sap.com/Protocols/SAPData" Version="1.0">
<edmx:DataServices xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
m:DataServiceVersion="1.0">
<Schema xmlns="http://schemas.microsoft.com/ado/2008/09/edm" Namespace="myNamespace" sap:schema-version="1">
<EntityType Name="System">
<Key>
<PropertyRef Name="Id" />
</Key>
<Property Name="Id" Type="Edm.Int32" Nullable="false" />
<Property Name="name" Type="Edm.String" sap:label="System Name" sap:creatable="false"
sap:updatable="false" sap:sortable="false" sap:required-in-filter="true"/>
<Property Name="description" Type="Edm.String" />
<Property Name="status" Type="Edm.String" />
<Property Name="type" Type="Edm.String" />
</EntityType>
<EntityContainer Name="ODataEntityContainer" m:IsDefaultEntityContainer="true">
<EntitySet Name="Systems" EntityType="myNamespace.System" />
<FunctionImport Name="NumberOfSystems" ReturnType="Collection(myNamespace.System)"
m:HttpMethod="GET" />
</EntityContainer>
</Schema>
</edmx:DataServices>
</edmx:Edmx>
I need to add following elements to above metatada
<Annotations Target="myNamespace.System"
xmlns="http://docs.oasis-open.org/odata/ns/edm">
<Annotation Term="com.sap.vocabularies.UI.v1.LineItem">
<Collection>
<Record Type="com.sap.vocabularies.UI.v1.DataField">
<PropertyValue Property="Value" Path="name" />
</Record>
<Record Type="com.sap.vocabularies.UI.v1.DataField">
<PropertyValue Property="Value" Path="description"/>
</Record>
<Record Type="com.sap.vocabularies.UI.v1.DataField">
<PropertyValue Property="Value" Path="status" />
</Record>
</Collection>
</Annotation>
</Annotations>
I came across the org.apache.olingo.commons.api.edm.provider.annotation package but cant find any suitable API. Please let me know how should I proceed.
Thanks in advance.
The annotations you would like to use have been introduced with OData V3 which is why they are not directly supported with the Olingo V2 library.
You can use the EdmProvider AnnotationElement and AnnotationAttribute classes to mimic this behaviour though. For example You can create a AnnotationElement with the name "Annotations" this element will then have the "AnnotationAttribute" Target=SomeString. Since an "AnnotationElement" can have child elements you can put your Collection element there. Namespaces are also handled with "AnnotationAttributes".
You can only attach the annotation to Edm elements which are derived from the EdmAnnotatable interface. So this is a difference to V3.
This is currently the only way to get this behaviour with Olingo V2.
I think the title of my question explains much of what I need. I am using the Data Importer Handler of Apache SOLR 5. I configured my solrconfig.xml, schema.xml and data-config.xml. It's working for now.
However, I need to add one more field. An Oracle Blob field. First, let me show my configurations:
data-config.xml
<dataConfig>
<!-- Datasource -->
<dataSource name="myDS"
setReadOnly="true"
driver="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:#//server.example.com:1521/service_name"
user="user"
password="pass"/>
<document name="products">
<entity name="product"
dataSource="myDS"
query="select * from products"
pk="id"
processor="SqlEntityProcessor">
<field column="id" name="id" />
<field column="name" name="name" />
<field column="price" name="price" />
<field column="store" name="store" />
<!-- I've added this blob field -->
<field column="picture" name="picture" />
</entity>
</document>
</dataConfig>
solrconfig.xml
<requestHandler name="/products" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
<!-- JDBCs -->
<lib dir="../../../lib" />
My fields in schema.xml
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="_version_" type="long" indexed="true" stored="true"/>
<field name="_text" type="string" indexed="true" stored="false" multiValued="true"/>
<field name="name" type="string" indexed="true" stored="true"/>
<field name="price" type="float" indexed="true" stored="true"/>
<!-- BLOB field -->
<field name="picture" type="binary" indexed="true" stored="true"/>
<copyField source="*" dest="_text"/>
<!-- ommited solr default fields -->
Now, when I start a full-importer, SOLR only indexes some records. This is the output after SOLR finish the importing:
Indexing completed. Added/Updated: 64 documents. Deleted 0 documents. (Duration: 04s)
Requests: 1 (0/s), Fetched: 1369 (342/s), Skipped: 0, Processed: 64 (16/s)
Started: less than a minute ago
As you can see, I have 1369 records, but SOLR only index 64 documents. If I remove the field picture from schema or, set index and stored attributes to false, SOLR import all documents.
I opened the SOLR log, and found this error when importing the blob field:
3436212 [Thread-19] WARN org.apache.solr.handler.dataimport.SolrWriter – Error creating document : SolrInputDocument(fields: [name=PRODUCTNAME, price=PRICE, store=STORE, picture=oracle.sql.BLOB#4130607a, _version_=1497915495941144576])
org.apache.solr.common.SolrException: ERROR: [doc=<ID>] Error adding field 'picture'='oracle.sql.BLOB#4130607a' msg=Illegal character .
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:176)
at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:78)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:931)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1085)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:697)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:71)
at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:263)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:511)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.lang.IllegalArgumentException: Illegal character .
at org.apache.solr.common.util.Base64.base64toInt(Base64.java:150)
at org.apache.solr.common.util.Base64.base64ToByteArray(Base64.java:117)
at org.apache.solr.schema.BinaryField.createField(BinaryField.java:89)
at org.apache.solr.schema.FieldType.createFields(FieldType.java:305)
at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:48)
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:123)
... 18 more
I checked querying directly against database, and it's working fine. I am using SOLR 5, ojdbc7 and Java 8. How can I use the binary field correctly in SOLR?
Update
I've changed the properties of picture in schema.xml setting indexed=false. This way:
<!-- BLOB field -->
<field name="picture" type="binary" indexed="false" stored="true"/>
Then, I restarted SOLR, reloaded my core, and did a Full-Import again. No success and same exception. The same 64 documents that I described above was imported and the field picture does not appear in JSON response. The query I execute is:
/select?q=*%3A*&wt=json&indent=true
I am working with liferay and want to store images in my MySQL database.
My service.xml goes like this:
<entity name="DLApp" local-service="true" remote-service="true">
<column name="blobId" type="long" primary="true" />
<column name="desc" type="String" />
<column name="image" type="Blob" />
</entity>
When i build services I get BUILD SUCCESSFUL message but an error in package com.test.blob.upload in the blobuploadModelImpl class.
How should I write function to store image in database and retrieve it?
I had same issue with Blob, so I changed it to String type, it was enought to store images up to 10 MB as string, I have used MySql as database and those column was mapped as LONGTEXT.
Also field should be 'hinted' in model hints as CLOB, See example below.
<entity name="UserExt" local-service="true" remote-service="true">
<column name="photo" type="String" />
</entity>
<model-hints>
<model name="com.project.model.UserExt">
<field name="photo" type="String">
<hint-collection name="CLOB" />
</field>
</model>
</model-hints>
Alternativelly you can store your images to Liferay's image gallery.
Suppose my entity name is SB_Claims so I generally get an error in SB_ClaimsModelImpl.java class
sb_ClaimsModelImpl._ContentBlobModel = null;
sb_ClaimsModelImpl variable is not declared or defined. We just declared this variable after sevice build operation. If you build service again then this code will remove by service builder. Always this code after service build and then do not build the service just deploy your portlet.
This will resolve the issue. In below I just modify that class.
#Override
public void resetOriginalValues()
{
SB_ClaimsModelImpl sb_ClaimsModelImpl=this;
sb_ClaimsModelImpl._ContentBlobModel = null;
}
Hope this will helpful for someone.
I am working with Solr 3.3.0 version and I need to index and search Images. I m able to index the image files but it fails to retrieve them as search results.
Can anyone of you help me out with this.
My data-config.xml is :
<dataConfig>
<dataSource type="BinFileDataSource" name="bin"/>
<document>
<entity name="f" processor="FileListEntityProcessor" recursive="true"
rootEntity="false"
dataSource="null" baseDir="C:/Files"
fileName=".*\.(DOC)|(PDF)|(XML)|(xml)|(JPEG)|(jpg)|(ZIP)|(zip)|(pdf)|(doc)" onError="skip">
<entity name="tika-test" processor="TikaEntityProcessor"
url="${f.fileAbsolutePath}" format="text" dataSource="bin" onError="skip">
<field column="Author" name="author" meta="true"/>
<field column="title" name="title" meta="true"/>
<field column="text" name="text"/>
<field column="id" name="id"/>
<field column="Keywords" name="keywords" meta="true"/>
</entity>
<field column="file" name="fileName"/>
<field column="fileAbsolutePath" name="links"/>
</entity>
</document>
</dataConfig>
This works fine for types other than images, I am not able to get images in search result
Ans) you have to search in solr admin page using the query in URL after indexed..
when ever you images are indexed it will display in Overview tab of solr admin ..
suppose it is not displaying the documents and time in solr web pag**e the image is not indexed still iam searching the same .....