I'm developing a demo and I'm stuck with this.
I want to list in a java web app all the attachments (PDFs for example), but a I am not able to retrieve and list them.
I'm only able to retrieve common data (String, Ints).
Is there a standard way to retrieve and show ?
I been reading all the posts but nothing seems to work.
Here is where I add the vendor, with the attachment:
public void addVendor(final Vendor vendor, final InputStream inputStream, final long size, final String contentType)
{
final Database db = getDb();
final int id = Integer.valueOf(vendor.get_id()) + 1;
final Response r1 = db.saveAttachment(inputStream, vendor.getName() + ".txt", contentType, String.valueOf(id), null);
vendor.setAttachment(r1);
final Response r = db.post(vendor);
System.out.println("Vendor created successfully. Id: " + r.getId() + ", rev: " + r.getRev());
System.out.println("File created successfully. Id: " + r1.getId() + ", rev: " + r1.getRev());
}
Here I where I try to retrive the data:
public List<Vendor> getAllVendors()
{
List<Vendor> Vendors = new ArrayList<Vendor>();
final List<Vendor> vend2 = new ArrayList<Vendor>();
//Get db
final Database db = getDb();
final InputStream s = null;
//Get all documents
Vendors = db.view("_all_docs").includeDocs(true).query(Vendor.class);
final Database db1 = getDb();
for (final Vendor vend : Vendors) {
final Response r1 = vend.getAttachment();
final int id = Integer.valueOf(vend.get_id()) + 1;
// Here I am look to the attachment with the _ID and _REV
final InputStream in = db1.find(r1.getId(), r1.getRev()); vend.setInput(in); vend2.add(vend);
}
return Vendors;
}
I this last code, I intended to create a new list with all my Vendor data plus the blob.
When I add the vendor ( in the first part ) , I saved the " response " of the attachement in the vendor object, SO when I tried to retrive I have the data to work with ( _id and _rev ) .
I'm assuming you want to list all documents that contain attachments. If so, you can create a MapReduce view similar to this:
function(doc) {
if (doc._attachments) {
emit(doc._id, null);
}
}
You would then call the view using something like this to get a list of document ids of documents that contain attachments:
GET /dbname/_design/designdocname/_view/docswithattachments
The above GET request would look something like this in Java:
List<Foo> list = db.view("designdocname/docswithattachments")
.query(Foo.class);
Related
Problem: My Android phone app can open various file types stored in an Android Room pre-populated SQLite database but it cannot open files the app itself has added to the pre-populated database (except it can open .txt files). I believe the issue is probably with how the I coded the copying and conversion of a selected file to byte[] data. The app is java based, and I have done this in Java before in a desktop app, so I just can't seem to find the issue. Maybe it is a permission issue, I'm just not sure and someone standing outside looking in may see what I can't.
What I have tried: Since the app can open various existing pre-populated files successfully from the DB, I've concentrated on and stepped through methods writing files to the DB. I'm not receiving any errors. I suspect it may just be minor issue since I can't seem to see it.
What I'm trying to do: I'm trying to emulate the desktop version of this app into a Android phone version. I know it's not recommended or common practice to populate files to a DB, but this app needs to be able to read and write files to the DB supporting it. This will be a full range of file types like the desktop version (e.g., pics, docs, audio, video, etc.). However, as I stated above, .txt files seem to have no issue. The user can select files stored on their phone into a table that captures the fileName and filePath to a TableRow in a TableLayout. Below are methods involved. The plan is to refactor functionality once I get it working:
Capturing the full path and filename for each row - Uses the captured filepath to convert to a byte[] to store the data. The filename and file byte data are stored in a Files table, example, Files(fileName, fileData(byte[])). Each file is added to an ArrayList<Files> which the method returns
public static List<Files> captureNoteFiles(TableLayout table){
List<Files> noteFiles = new ArrayList<>();
int i = table.getChildCount();
if(i>1){
for (int itr = 1; itr<i; itr++) { // iterating through indexes
TableRow tr = (TableRow) table.getChildAt(itr);
TextView tv = (TextView) tr.getChildAt(1); // 1 is the file path position
File f = new File(tv.getText().toString());
String n = f.getName();
try {
FileInputStream fis = new FileInputStream(f.getPath());
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
for (int read; (read = fis.read(buf)) != -1; ) {
bos.write(buf, 0, read);
}
fis.close();
noteFiles.add(new Files(0, n, bos.toByteArray()));
} catch (Exception e) {
e.printStackTrace();
Log.d("Input File", e.toString());
}
}
}
return noteFiles;
}
Iteration of the ArrayList - The ArrayList<Files> is iterated and populated to the Files table and an ID capture to associate those files with a particular note of reference.
public static void addNewNoteFiles(int noteID, List<Files> nf){
if(nf.size()>0) {
for (Files f : nf) {
long id = rdb.getFilesDao().addFile(f);
rdb.getFilesByNoteDao().insert(new FilesByNote(noteID, (int) id));
}
}
}
Files Entity
#Entity(tableName = "Files")
public class Files implements Parcelable {
#PrimaryKey(autoGenerate = true)
#ColumnInfo(name = "FileID")
private int fileID;
#ColumnInfo(name = "FileName")
private String fileName;
#TypeConverters(FileTypeConverter.class)
#ColumnInfo(name = "FileData", typeAffinity = ColumnInfo.TEXT)
private byte[] fileData;
#SuppressWarnings(RoomWarnings.CURSOR_MISMATCH)
public Files(int fileID, String fileName, byte[] fileData){
this.fileID = fileID;
this.fileName = fileName;
this.fileData = fileData;
}
}
First you are assuming that an insert works as per :-
long id = rdb.getFilesDao().addFile(f);
rdb.getFilesByNoteDao().insert(new FilesByNote(noteID, (int) id));
What if the row isn't inserted? and returns an id of -1?
So I'd suggest adding getters to the Files class such as :-
public int getFileID() {
return fileID;
}
public String getFileName() {
return fileName;
}
public byte[] getFileData() {
return fileData;
}
and then add the following to FilesDao :-
#Query("SELECT coalesce(length(FileData)) FROM Files WHERE FileID=:fileId")
abstract long getFilesDataLength(long fileId);
and then amending the addNewNoteFiles to be :-
public static void addNewNoteFiles(int noteID, List<Files> nf){
final String TAG = "ADDNEWNOTE";
if(nf.size()>0) {
for (Files f : nf) {
long id = rdb.getFilesDao().addFile(f);
if (id > 0) {
long lengthOfFileData = rdb.getFilesDao().getFilesDataLength(id);
Log.d(TAG,
"Inserted File = " + f.getFileName() +
" DataLength = " + f.getFileData().length +
" ID = " + f.getFileID() +
" Length of Stored Data = " + lengthOfFileData);
if (f.getFileData().length != lengthOfFileData) {
Log.d(TAG,"WARNING FileData length MISMATCH for File = " + f.getFileName() + "\n\t Expected " + f.getFileData().length + " Found " + lengthOfFileData);
}
rdb.getFilesByNoteDao().insert(new FilesByNote(noteID, (int) id));
} else {
Log.d(TAG,"NOT INSERTED File = " + f.getFileName());
}
}
}
}
Run and check the log. Are all the files inserted? Do the lengths match? Are the lengths as expected (if all 0 lengths, or some, then obviously something is amiss when building the ByteArrayOutputStream)
You may wish to add similar for inserting the FilesByNote i.e. have the insert Dao return a long (it returns the rowid) and check if the value is > 0.
You may wonder what rowid is. Well it's a normally hidden column, perhaps hidden as it would appear that FilesByNotes is an associative table mapping(associating) Note(s) with Files and as such has a composite primary key NoteId and FileId which is not an alias of the rowid, so rowid will be hidden as such. However, the value will be auto-generated or -1 if no row is inserted.
ALL tables, with the exception of tables defined with WITHOUT ROWID, have a rowid column. Room does not allow thee definition of WITHOUT ROWID tables.
You wouldn't be concerned about the value if it's greater than 0, just that it is greater than 0 and thus a row was inserted.
The above may help to determine any issues encountered when inserting the data. If there are none found then the issue is else where.
I am using Spring Boot, Snowflake and AWS S3.
I have SQL query that queries two tables and gets the result. That result I have to write to S3 like CSV file and get URL for download in return.
I am doing that by creating temporary table and deleting it after data is copied to S3.
Here is code I have:
#Override
public void getUserTest(String userId) {
String q = "CREATE TEMPORARY TABLE \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" AS SELECT \"ID\", \"FIRST_NAME\", \"LAST_NAME\" from \"TEST\".\"PUBLIC\".\"USER_TABLE\"\n" +
" where \"ID\" = ?\n" +
" union all\n" +
" select \"ID\",\"ACCOUNT_NAME\", \"ACCOUNT_NUMBER\" from \"TEST\".\"PUBLIC\".\"ACCOUNT_DATA\"\n" +
" where \"ID\" = ?";
jdbcTemplate.query(q, s -> {}, userId, userId);
}
Method that writes to S3.
#Override
public URL writeToS3() {
String q = "copy into s3://snowflake171 from \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST;\n";
jdbcTemplate.query(q, s -> {});
URL url = generateURL();
String dropTable = "drop table if exists \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\"";
jdbcTemplate.query(dropTable, s -> {});
return url;
}
Method that generated URL:
public URL generateURL() {
try {
BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKey, secretKey);
final AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new
AWSStaticCredentialsProvider(awsCreds)).withRegion(clientRegion).build();
// Set the presigned URL to expire after 2h.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = Instant.now().toEpochMilli();
expTimeMillis += 1000 * 60 * 120;
expiration.setTime(expTimeMillis);
// Generate the presigned URL.
System.out.println("Generating pre-signed URL.");
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucket, objectKey)
.withMethod(HttpMethod.GET)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println("Pre-Signed URL: " + url.toString());
return url;
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
return null;
}
Data is queried by userId I sent in. Everything works fine, but I am generating every time file with same name. And if I dont delete existing file in S3 I am not able to upload new one.
I should be able to upload different files for different userId.
How can I do this?
How can I give different names from files created in S3?
I have seen this in docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html but I don't know the best way to apply in code.
Is there a way I can add userId as prefix to filename?
You can add a custom object key with userId in your COPY query :
#Override
public URL writeToS3(userId) {
String q = "copy into s3://snowflake171/" + userId + " from \"TEST\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST;\n";
jdbcTemplate.query(q, s -> {});
I need get an object like OLAPDataSetInterface but without using an OLAP server.
I used to use:
//Connect to OLAP server
OLAPDataSetInterface cube = objectInSession.getOlapDataSet();
//Get the info
cube.execute("query_mdx");
I tried use DataSetInterface but doesnt work:
com.sas.sasserver.dataset.DataSetInterface ds = null;
//Getting my temporary table
ds.setDataSet("WORK.my_table");
And i do the following:
//BBDD connector
WorkspaceConnector connector = factory.getWorkspaceConnector(0L);
IWorkspace workspace = connector.getWorkspace();
ILanguageService ls = workspace.LanguageService();
//This creates my temporary table in the library WORK (WORK.my_table)
String stmt = "%include \"/saswork/MY_PROGRAM.sas\" ;";
ls.Submit(stmt);
com.sas.sasserver.dataset.DataSetInterface ds = null;
//ds = ...
This is in C# but hould help understand the dataset retrieval process. I don't work with SAS OLAP so cannot tellyou how to work that item.
public string GetDataSet(string sasDirectory, string dataset)
{
Common.Log($"Getting SAS dataset ({dataset}) at {sasDirectory}");
DataTable dt = new DataTable(dataset);
try
{
using (var cn = new OleDbConnection($#"Provider=SAS.LocalProvider; Data Source={sasDirectory}"))
{
cn.Open();
var cmd = cn.CreateCommand();
cmd.CommandType = CommandType.TableDirect;
cmd.CommandText = dataset;
var sas = new OleDbDataAdapter(cmd);
var ds = new System.Data.DataSet();
sas.Fill(ds, dataset);
dt = ds.Tables[0];
Common.Log($"SAS dataset loaded.");
}
}
catch (Exception ex)
{
string errMessage = "Unable to get the SAS dataset. Library: " + sasDirectory + ", DataSet: " + dataset + ", " +
ex.TargetSite.Name;
Common.Log($"SAS Error in {MethodBase.GetCurrentMethod().Name}", MessageType.Error, ex);
}
return dt.ToCsv('\t');
}
My XPage gathers information which I use to populate a document in a different Domino database. I use a link button (so I can open another XPage after submission). The onClick code is as follows:
var rtn = true
var util = new utilities()
var hostURL = configBean.getValue("HostURL");
var userAttachment;
//set up info needed for checking duplicates
var attachName=getComponent("attachmentIdentifier").getValue();
var serialNbr = getComponent("serialNumber").getValue();
userAttachment = user+"~"+attachName;
var userSerial = user+"~"+serialNbr;
//Done setting info needed
//check for duplicates
rtn = utilBean.checkAttachmentName(userAttachment, userSerial)
//done
if(rtn==true){
var doc:Document = document1;
dBar.info("ALL IS GOOD");
var noteID:String=document1.getNoteID();
dBar.info("Calling saveNewAttachment using NoteID " + noteID )
rtn=utilBean.saveNewAttachment(session,noteID ); //<<< I get error here
dBar.info("rtn = " + rtn)
return "xsp-success";
view.postScript("window.open('"+sessionScope.nextURL+"')")
}else if (rtn==false){
errMsgArray = utilBean.getErrorMessages();
for(err in errMsgArray){
//for (i=0; i < errMsgArray.size(); i++){
dBar.info("err: "+ err.toString());
if (err== "nameUsed"){
//send message to XPXage
facesContext.addMessage(attachmentIdentifier.getClientId(facesContext) , msg(langBean.getValue("duplicateName")));
}
if(err=="serialUsed"){
//send message to XPXage
facesContext.addMessage(serialNumber.getClientId(facesContext) , msg(langBean.getValue("duplicateSerial")));
}
}
return "xsp-failure";
}
And the java code that delivers the error is this
public boolean saveNewAttachment(Session ses, String noteID)
throws NotesException {
debugMsg("Entering saveNewAttachment and NOTEID = "+noteID);
// this is used when the user saves an attachment to to the
// user profiles db
boolean rtn = false;
Document doc;
ConfigBean configBean = (ConfigBean)
ExtLibUtil.resolveVariable(FacesContext.getCurrentInstance(),
"configBean");
String dbName = (String) configBean.getValue("WebsiteDbPath");
debugMsg("A");
Database thisDB = ses.getDatabase(ses.getServerName(), dbName, false);
String value;
try {
debugMsg("noteID: "+noteID);
The next line throws the NotesException error
doc = thisDB.getDocumentByID("noteID");
debugMsg("C");
} catch (Exception e) {
debugMsg("utilitiesBean.saveAttachment: " + e.toString());
e.printStackTrace();
System.out.println("utilitiesBean.saveAttachment: " + e.toString());
throw new RuntimeException("utilitiesBean.saveAttachment: "
+ e.toString());
}
return rtn;
}
I might be going about this wrong. I want to save the document which the data is bound to the User Profile database but if I submit it I need to redirect it to a different page. That is why I am using a link, however, I am having a hard time trying to get the document saved.
Has document1 been saved before this code is called? If not, it's not in the backend database to retrieve via getDocumentByID().
I'm assuming this line has been copied into here incorrectly, because "noteID" is not a NoteID or a variable holding a NoteID, it's a string.
doc = thisDB.getDocumentByID("noteID");
I'm using Java to download HTML contents of websites whose URLs are stored in a database. I'd like to put their HTML into database, too.
I'm using Jsoup for this purpose:
public String downloadHTML(String byLink) {
String htmlInPage = "";
try {
Document doc = Jsoup.connect(byLink).get();
htmlInPage = doc.html();
} catch (org.jsoup.UnsupportedMimeTypeException e) {
// process this and some other exceptions
}
return htmlInPage;
}
I'd like to download websites concurrently and use this function:
public void downloadURL(int websiteId, String url,
String categoryName, ExecutorService executorService) {
executorService.submit((Runnable) () -> {
String htmlInPage = downloadHTML(url);
System.out.println("Category: " + categoryName + " " + websiteId + " " + url);
String insertQuery =
"INSERT INTO html_data (website_id, html_contents) VALUES (?,?)";
dbUtils.query(insertQuery, websiteId, htmlInPage);
});
}
dbUtils is my class based on Apache Commons DbUtils. Details are here: http://pastebin.com/iAKXchbQ
And I'm using everything mentioned above in a such way: (List<Object[]> details are explained on pastebin, too)
public static void main(String[] args) {
DbUtils dbUtils = new DbUtils("host", "db", "driver", "user", "pass");
List<String> categoriesList =
Arrays.asList("weapons", "planes", "cooking", "manga");
String sql = "SELECT lw.id, lw.website_url, category_name " +
"FROM list_of_websites AS lw JOIN list_of_categories AS lc " +
"ON lw.category_id = lc.id " +
"where category_name = ? ";
ExecutorService executorService = Executors.newFixedThreadPool(10);
for (String category : categoriesList) {
List<Object[]> sitesInCategory = dbUtils.select(sql, category );
for (Object[] entry : sitesInCategory) {
int websiteId = (int) entry[0];
String url = (String) entry[1];
String categoryName = (String) entry[2];
downloadURL(websiteId, url, categoryName, executorService);
}
}
executorService.shutdown();
}
I'm not sure if this solution is correct but it works. Now I want to modify code to save HTML not from all websites in my database, but only their fixed ammount in each category.
For example, download and save HTML of 50 websites from the "weapons" category, 50 from "planes", etc. I don't think it's necessary to use sql for this purpose: if we select 50 sites per category, it doesn't mean we save them all, because of possibly incorrect syntax and connection problems.
I've tryed to create separate class implementing Runnable with fields: counter and maxWebsitesPerCategory, but these variables aren't updated. Another idea was to create field Map<String,Integer> sitesInCategory instead of counter, put each category as a key there and increment its value until it reaches maxWebsitesPerCategory, but it didn't work, too. Please, help me!
P.S: I'll also be grateful for any recommendations connected with my realization of concurrent downloading (I haven't worked with concurrency in Java before and this is my first attempt)
How about this?
for (String category : categoriesList) {
dbUtils.select(sql, category).stream()
.limit(50)
.forEach(entry -> {
int websiteId = (int) entry[0];
String url = (String) entry[1];
String categoryName = (String) entry[2];
downloadURL(websiteId, url, categoryName, executorService);
});
}
sitesInCategory has been replaced with a stream of at most 50 elements, then your code is run on each entry.
EDIT
In regard to comments. I've gone ahead and restructured a bit, you can modify/implement the content of the methods I've suggested.
public void werk(Queue<Object[]> q, ExecutorService executorService) {
executorService.submit(() -> {
try {
Object[] o = q.remove();
try {
String html = downloadHTML(o); // this takes one of your object arrays and returns the text of an html page
insertIntoDB(html); // this is the code in the latter half of your downloadURL method
}catch (/*narrow exception type indicating download failure*/Exception e) {
werk(q, executorService);
}
}catch (NoSuchElementException e) {}
});
}
^^^ This method does most of the work.
for (String category : categoriesList) {
Queue<Object[]> q = new ConcurrentLinkedQueue<>(dbUtils.select(sql, category));
IntStream.range(0, 50).forEach(i -> werk(q, executorService));
}
^^^ this is the for loop in your main
Now each category tries to download 50 pages, upon failure of downloading a page it moves on and tries to download another page. In this way, you will either download 50 pages or have attempted to download all pages in the category.