We Keep Coding

Transfer file from SFTP to ADLS

We are currently in the process of exploring the sshj library to download a file from SFTP path into ADLS. We are using the example as reference.
We have already configured the ADLS Gen2 storage in Databricks to be accessed as an abfss URL.
We are using scala within Databricks.
How should we pass the abfss path as FileSystemFile object in the get step ?
sftp.get("test_file", new FileSystemFile("abfss://<container_name>#a<storage_account>.dfs.core.windows.net/<path>"));
Is the destination supposed to be a file path only or file path with file name?

Use streams. First obtain InputStream of the source SFTP file:
RemoteFile f = sftp.open(sftpPath);
InputStream is = f.new RemoteFileInputStream(0);
(How to read from the remote file into a Stream?)
Then obtain OutputStream of the destination file on ADLS:
OutputStream os = adlsStoreClient.createFile(adlsPath, IfExists.OVERWRITE);
(How to upload and download a file from my locale to azure adls using java sdk?)
And copy from the first to the other:
is.transferTo(os);
(Easy way to write contents of a Java InputStream to an OutputStream)

insert a file into postgres

How do I insert a file that resides in windows in c:\temp\sample.txt
I have tried
insert into lob_file VALUES (5, pg_read_file('c://temp//sample.txt')::BYTEA);
and got
ERROR: could not start file c://temp//sample.txt. No such file or
directory.

Is data base running locally or on a Windows server and you have moved the file to the server. See Docs
The functions shown in Table 9.94 provide native access to files on
the machine hosting the server. Only files within the database cluster
directory and the log_directory can be accessed unless the user is
granted the role pg_read_server_files. Use a relative path for files
in the cluster directory, and a path matching the log_directory
configuration setting for log files.

How to download file to local machine instead of cloud desktop in Java?

I have a file on S3 which I am downloading using a s3 handler. It gets downloaded to the cloud desktop (where my code is located). I have used the following to get the path:
String home = System.getProperty("user.home");
File file = new File(home+"/Downloads/" + fileName + ".txt");
I want the file to be downloaded to the local machine instead of the cloud desktop. Is there a way to route the file from cloud desktop to local? What can be done to get the file on the local machine instead? Any ideas?

Backup and restore of Hsqldb database in java code

I am new in Hsqldb database. I want to know how to take backup and restore of Hsqldb database through java code.

Use the BACKUP DATABASE TO command.
Here is a link to the documentation:
HSQLDB System Management Documentation
I haven't tested this, but I imagine it's something along the lines of:
String backup = "BACKUP DATABASE TO " + "'" + filePath + "' BLOCKING";
PreparedStatement preparedStatement = connection.prepareStatement(backup);
preparedStatement.execute();
You'll want to wrap it in a try-catch block of course.
As far as restoring the db goes, I think you have to perform that while the database is offline using the DbBackupMain application. So you would issue this command at the command line:
java -cp hsqldb.jar org.hsqldb.lib.tar.DbBackupMain --extract tardir/backup.tar dbdir

Each HyperSQL database is called a catalog. There are three types of catalog depending on how the data is stored.
Types of catalog data :
mem: stored entirely in RAM - without any persistence beyond the JVM process's life
file: stored in filesystem files
res: stored in a Java resource, such as a Jar and always read-only
To back up a running catalog, obtain a JDBC connection and issue a BACKUP DATABASE command in SQL. In its most simple form, the command format below will backup the database as a single .tar.gz file to the given directory.
BACKUP DATABASE TO <directory name> BLOCKING [ AS FILES ]
The directory name must end with a slash to distinguish it as a directory, and the whole string must be in single quotes like so: 'subdir/nesteddir/'.
To back up an offline catalog, the catalog must be in shut down state. You will run a Java command like
java -cp hsqldb.jar org.hsqldb.lib.tar.DbBackupMain --save tardir/backup.tar dbdir/dbname
. In this example, the database is named dbname and is in the dbdir directory. The backup is saved to a file named backup.tar in the tardir directory.
where tardir/backup.tar is a file path to the *.tar or *.tar.gz file to be created in your file system, and dbdir/dbname is the file path to the catalog file base name.
You use DbBackup on your operating system command line to restore a catalog from a backup.
java -cp hsqldb.jar org.hsqldb.lib.tar.DbBackupMain --extract tardir/backup.tar dbdir
where tardir/backup.tar is a file path to the *.tar or *.tar.gz file to be read, and dbdir is the target directory to extract the catalog files into. Note that dbdir specifies a directory path, without the catalog file base name. The files will be created with the names stored in the tar file.
For more details refer

So in java + SPring + JdbcTemplate
Backup (On-line):
#Autowired
public JdbcTemplate jdbcTemplate;
public void mainBackupAndRestore() throws IOException {
...
jdbcTemplate.execute("BACKUP DATABASE TO '" + sourceFile.getAbsolutePath() + "' BLOCKING");
}
This will save .properties, .scripts and .lobs file to a tar in sourceFile.getAbsolutePath().
Restore:
DbBackupMain.main(new String[] { "--extract", baseDir.getAbsolutePath(),
System.getProperty("user.home") + "/restoreFolder" });
This will get files from baseDir.getAbsolutePath() and will put them in userHome/restoreFolder where you can check if all restore is OK.
lobs contains lob/blob data, scripts contains executed queries.

File upload from a web-based application to a linux server

It seems that my code won't work after learning that the machine that I'll be pointing the upload path to is a Linux box.
My use case is, a user logs in to the web app, chooses a file to upload, then click upload button. Is it possible to do this direct from the Java code to the Linux server using appropriate ssh or scp libraries if there is any?
EDIT: Here's my current code.
#Override
public void fileTransfer(File uploadedFile, String fileName, String pathTemp) {
File destFile = new File( pathTemp + File.separator + fileName);
try{
FileUtils.copyFile(uploadedFile, destFile);
String getTempFile = destFile.toString();
String tempPath = getTempFile.replace("\\", "\\\\");
File tempFile = new File(tempPath); // 1st file
String tempFileName = tempFile.getName();
String fileSave = getUploadPathSave().replace("\\", "\\\\");
tempFile.renameTo(new File(fileSave + tempFileName));
} catch (IOException ex) {
System.out.println("Could not copy file " + fileName);
ex.printStackTrace();
}
}

If your app is deployed at one place only (not mass distribution), the easiest way would be:
create samba share on linux machine
map samba share to logical drive on windows machine
do usual file copy with java functions.
attention: renameTo will not work between drives. You'll need to copy input stream to output stream or, better, use apache commons-io functions for that.

There are different possibilities:
If you can create a shared directory in linux and mount it under windows (see Samba. Then you can write to that directory like a local directory. File will go to the linux server.
Use a library like Jsch to upload the file from windows server to linux server.

There are certain things you can do:
1-> If you can program your linux server, then you can make a program that listens to user request on a port, and stores data in file. Then you can send you files to that port of server.
2-> The other way is you can use some sort of script to create ssh-connection to server and then you can just add file through ssh, but here your java program will not be useful.
I personally use my own program to share files between 2 machines in same network.
You can use it,if it will be useful for you: https://github.com/RishabhRD/xshare