We Keep Coding

Running ffprobe on file lying in Azure Blob storage

I am trying to get metadata of a file lying in Azure blob storage.
I am using ffprobe for this purpose. Though it works, since the ffprobe binary lies on my local system and file lies in Blob, the entire process is too slow
What would be the best way to do the above, getting meta data for a remote file?

Two ways for your reference:
1.Use blob.downloadAttributes(),then use blob.getMetadata()
This method populates the blob's system properties and user-defined
metadata. Before reading or modifying a blob's properties or metadata,
call this method or its overload to retrieve the latest values for the
blob's properties and metadata from the Microsoft Azure storage
service.
2.Use get-metadata-activity in ADF.
Get a file's metadata:

How to compress files on azure data lake store

I'm using Azure data lake store as a storage service for my Java app, sometimes I need to compress multiples files, what I do for now is I copy all files into the server compress them locally and then send the zip to azure, even though this is work it take a lot of time, so I'm wondering is there a way to compress files directly on azure, I checked the data-lake-store-SDK, but there's no such functionality.

Unfortunately, at the moment there is no option to do that sort of compression.
There is an open feature request HTTP compression support for Azure Storage Services (via Accept-Encoding/Content-Encoding fields) that discusses uploading compressed files to Azure Storage, but there is no estimation on when this feature might be released.
The only option for you is to implement such a mechanism on your own (using an Azure Function for example).
Hope it helps!

Compression in google bigquery does not have extension in the generated file

I am trying to run a query in Google Big Query and export the data to Google cloud storage using GZIP compression.
JobConfigurationExtract jobExtractConfig = new JobConfigurationExtract().setSourceTable(tableReference).set("results.csv", "CSV")
.setDestinationUri("gs://dev-app-uploads/results.zip")
.setCompression("GZIP");
By using this config i am able to generate a results.zip file successfully in cloud storage in the configured bucket dev-app-uploads. But the file inside the zip is generated without a .csv extension. When i extract the zip file, i am getting a "results" file and when i manually add the extention .csv and open the file, the contents are there.
But my necessity is to generate the file with .csv extension and zip it and place it in cloud storage.
Please let me know if this is possible or any other better options to upload data from big query using compression.

Instead of
gs://dev-app-uploads/results.zip
use below
gs://dev-app-uploads/results.csv.zip

Play Framework file upload to in memory to S3

I am writing a web server with Play framework 2.6 in Java. I want to upload a file to WebServer through a multipart form and do some validations, then upload the file s3. The default implementation in play saves the file to a temporary file in the file system but I do no want to do that, I want to upload the file straight to AWS S3.
I looked into this tutorial, which explains how to save file the permanently in file system instead of using temporary file. To my knowledge I have to make a custom Accumulator or a Sink that saves the incoming ByteString(s) to a byte array? but I cannot find how to do so, can someone point me in the correct direction?
thanks

Zipping a directory to a remote location in Java

I'm trying to create a zip file from a directory in Java, then place that file on a remote server (a network share). I'm currently getting an error because the Java File object cannot reference a remote location and I'm not really sure where to go from there.
Is there a way to zip a directory in Java to a remote location without using the File class?

Create the ZIP file locally and use either commons-net FTP or SFTP to move the ZIP file across to the remote location, assuming that by "remote location" you mean some FTP server, or possibly a blade on your network.
If you are using the renameTo method on java.io.File, note that this doesn't work on some operating systems (e.g. Solaris) where the locations are on different shares. You would have to do a manual copy of the file data from one location to another. This is pretty simple using standard Java I/O.