i'm getting my hands on Drools ( with java ) for the first time and i'm quite confused about it's sessions and ability to work with collections of objects.
this is the case:
i'm building a web-application made of rest-services.
i have a class called Log with two fields ( eventType and RiskLevelId ).
Mycode retrieves from a db several objects of this kind in a defined time frame.
If this collection of objects happens to contain one Log with eventType == 2 and RiskLevelId == 1 and another Log with eventType == 3 and RiskLevelId == 1, the rule should be executed.
Via Drools interfaces I correctly retrieve KieServices, KieBuilder, KieContaier, KieBase and KieSession.
try {
// load up the knowledge base
KieServices kieServices = KieServices.Factory.get();
KieFileSystem kfs = kieServices.newKieFileSystem();
FileInputStream fis = f;
kfs.write( "src/main/resources/simple.drl",
kieServices.getResources().newInputStreamResource( fis ) );
KieBuilder kieBuilder = kieServices.newKieBuilder( kfs ).buildAll();
Results results = kieBuilder.getResults();
if( results.hasMessages( Message.Level.ERROR ) ){
System.out.println( results.getMessages() );
throw new IllegalStateException( "### errors ###" );
}
KieContainer kieContainer = kieServices.newKieContainer( kieServices.getRepository().getDefaultReleaseId() );
KieBase kieBase = kieContainer.getKieBase();
kieSession = kieContainer.newKieSession();
}catch (Throwable t) {
t.printStackTrace();
}
i then retrieve each single Log istance in a for loop. staying in the loop i also add the object to the KieSession and fire the rule:
#Autowired
KieSessionFactory kieSessionFactory;
#Override
public void run() {
KieSession kieS = kieSessionFactory.getKieSessionCheckSavedLog();
try {
List<Log> logs = logRepo.getAllInGivenTimeSec(10);
for(Log l : logs) {
kieS.insert(l);
kieS.fireAllRules();
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Here comes the rule i've written:
package com.sample
import it.protodrools.beans.Log;
dialect "java"
rule "log2"
when
$l1 : Log( eventType == 2 && riskLevelId == 1);
$l2 : Log( this != $l1 && eventType == 3 && riskLevelId == 1 );
then
System.out.println( "deadly threat !" );
end
My question is: will this rule take in account the whole list of objects that i'm passing ( though not via List, as i've read this is not a good practice ) and thus consider whether there's a condition-matching pair of object among those i'v passed ?
woukd you suggest some different workaround ?
thanks in advance
No, it will not.
for(Log l : logs) {
kieS.insert(l);
kieS.fireAllRules();
}
According to your loop you will insert an object and after each insert immediately afterwards you fire all rules. I am not sure how Drools will react to your loop, but what you probably want to do is insert all Logs in the working memory and then fire the rules:
for(Log l : logs) {
kieS.insert(l);
}
kieS.fireAllRules();
Designing a JUnit test class would show you this immediately though.
Update 4 - rephrasing question for clarity
I am using Pull Queues to feed back-end workers tasks that send push notifications. I can see the front-end instance queue the task in the logs. However, the task is only occasionally handled by the back-end. I see no indication of why the task disappears prior to being handled and deleted from the queue.
This may be related: I am seeing an unusually high number of TransientFailureExceptions when attempting to lease tasks from the queue - despite sleeping between attempts.
Everything works properly on my development server (and an earlier version had worked in production) but production is no longer working properly. At first I thought it was a certificate issue. However, notifications are sometimes sent when the backend first starts.
There is no indication that an error is happening except for the TransientFailureException when I call leaseTasks on the queue. Also, it seems to take a very long time for my logs to show up.
I can provide more information and code snippets as needed.
Thanks for the help.
Update 1:
The application uses 10 pull queues. It would normally use 2 but queue tagging is still considered experimental. They are declared in the standard fashion:
<queue>
<name>gcm-henchdist</name>
<mode>pull</mode>
</queue>
The lease tasks function is:
public boolean processBatchOfTasks()
{
List< TaskHandle > tasks = attemptLeaseTasks();
if( null == tasks || tasks.isEmpty() )
{
return false;
}
processLeasedTasks( tasks );
return true;
}
private List< TaskHandle > attemptLeaseTasks()
{
for( int attemptNnum = 1; !LifecycleManager.getInstance().isShuttingDown(); ++attemptNnum )
{
try
{
return m_taskQueue.leaseTasks( m_numLeaseTimeUnits, m_leaseTimeUnit, m_maxTasksPerLease );
} catch( TransientFailureException exc )
{
LOG.warn( "TransientFailureException when leasing tasks from queue '{}'", m_taskQueue.getQueueName(), exc );
ApiProxy.flushLogs();
} catch( ApiDeadlineExceededException exc )
{
LOG.warn( "ApiDeadlineExceededException when when leasing tasks from queue '{}'",
m_taskQueue.getQueueName(), exc );
ApiProxy.flushLogs();
}
if( !backOff( attemptNnum ) )
{
LOG.warn( "Failed to lease tasks." );
break;
}
}
return Collections.emptyList();
}
where the lease variables are 30, TimeUnit.MINUTES, 100 respectively
the processBatchOfTasks function is polled via:
private void startPollingForClient( EClientType clientType )
{
InterimApnsCertificateConfig config = InterimApnsCertificateConfigMgr.getConfig( clientType );
Queue notificationQueue = QueueFactory.getQueue( config.getQueueId().getName() );
ApplePushNotificationWorker worker = new ApplePushNotificationWorker(
notificationQueue,
m_messageConverter.getObjectMapper(),
config.getCertificateBytes(),
config.getPassword(),
config.isProduction() );
LOG.info( "Started worker for {} polling queue {}", clientType, notificationQueue.getQueueName() );
while ( !LifecycleManager.getInstance().isShuttingDown() )
{
boolean tasksProcessed = worker.processBatchOfTasks();
ApiProxy.flushLogs();
if ( !tasksProcessed )
{
// Wait before trying to lease tasks again.
try
{
//LOG.info( "Going to sleep" );
Thread.sleep( MILLISECONDS_TO_WAIT_WHEN_NO_TASKS_LEASED );
//LOG.info( "Waking up" );
} catch ( InterruptedException exc )
{
LOG.info( "Polling loop interrupted. Terminating loop.", exc );
return;
}
}
}
LOG.info( "Instance is shutting down" );
}
and the thread is created via:
Thread thread = ThreadManager.createBackgroundThread( new Runnable()
{
#Override
public void run()
{
startPollingForClient( clientType );
}
} );
thread.start();
GCM notifications are handled in a similar fashion.
Update 2
The following is the backoff function. I have verified in the logs (with both GAE and my own timestamps) that the sleep is incrementing properly
private boolean backOff( int attemptNo )
{
// Exponential back off between 2 seconds and 64 seconds with jitter
// 0..1000 ms.
attemptNo = Math.min( 6, attemptNo );
int backOffTimeInSeconds = 1 << attemptNo;
long backOffTimeInMilliseconds = backOffTimeInSeconds * 1000 + (int)( Math.random() * 1000 );
LOG.info( "Backing off for {} milliseconds from queue '{}'", backOffTimeInMilliseconds, m_taskQueue.getQueueName() );
ApiProxy.flushLogs();
try
{
Thread.sleep( backOffTimeInMilliseconds );
} catch( InterruptedException e )
{
return false;
}
LOG.info( "Waking up from {} milliseconds sleep for queue '{}'", backOffTimeInMilliseconds, m_taskQueue.getQueueName() );
ApiProxy.flushLogs();
return true;
}
Update 3
The tasks are added to the queue within a transaction on a front-end instance:
if( null != queueType )
{
String deviceName;
int numDevices = deviceList.size();
for ( int iDevice = 0; iDevice < numDevices; ++iDevice )
{
deviceName = deviceList.get( iDevice ).getName();
LOG.info( "Queueing Your-Turn notification for user: {} device: {} queue: {}", user.getId(), deviceName, queueType.getName() );
Queue queue = QueueFactory.getQueue( queueType.getName() );
queue.addAsync( TaskOptions.Builder.withMethod( TaskOptions.Method.PULL )
.param( "alertLocKey", "NOTIF_YOUR_TURN" ).param( "device", deviceName ) );
}
}
I know that the transaction succeeds because the database updates correctly.
In the logs I see the "Queuing Your-Turn notification..." entry, but I see nothing appear on the back-end logs.
In the administration panel, I see Task Queue API Calls increment by 1 as well as Task Queue Stored Task Count increment by 1. However, the queue that was written to shows zero in both the Tasks In Queue and Leased In Last Minute fields.
The TransientFailureException JavaDoc says that "The requested operation may succeed if attempted again" (because the failure is transient). Therefore when this exception is thrown your code should loop back and repeat the leaseTasks call. Furthermore AppEngine does not have to redo the request itself because it notified you via the exception that you should do so.
It's a pity you repeat the method name leaseTasks as one of your own because now it's not clear which one I'm referring to when I mention leaseTasks. Still, wrap the inner call to m_taskQueue.leaseTasks in a while loop and an additional try block to catch only the TransientFailureException. Use a flag to end the while loop only if that exception is not thrown.
Is that enough explanation, or do you need a complete source code listing?
It appears that the culprit may have been that I was calling addAsync when enqueuing the task instead of just calling add.
I replaced the call and things seem to be consistently working now. I would like to know why this makes a difference and will update the answer when I find the reason.
My desktop application, written in java, tries to download public files from Google Drive. As i found out, it can be implemented by using file's webContentLink (it's for ability to download public files without user authorization).
So, the code below works with small files:
String webContentLink = aFile.getWebContentLink();
InputStream in = new URL(webContentLink).openStream();
But it doesn't work on big files, because in this case file can't be downloaded directly via webContentLink without user confirmation with google virus scan warning. See an example: web content link.
So my question is how to get content of a public file from Google Drive without user authorization?
Update December 8th, 2015
According to Google Support using the
googledrive.com/host/ID
method will be turned off on Aug 31st, 2016.
I just ran into this issue.
The trick is to treat your Google Drive folder like a web host.
Update April 1st, 2015
Google Drive has changed and there's a simple way to direct link to your drive. I left my previous answers below for reference but to here's an updated answer.
Create a Public folder in Google Drive.
Share this drive publicly.
Get your Folder UUID from the address bar when you're in that folder
Put that UUID in this URL
https://googledrive.com/host/<folder UUID>/
Add the file name to where your file is located.
https://googledrive.com/host/<folder UUID>/<file name>
Which is intended functionality by Google
new Google Drive Link.
All you have to do is simple get the host URL for a publicly shared drive folder. To do this, you can upload a plain HTML file and preview it in Google Drive to find your host URL.
Here are the steps:
Create a folder in Google Drive.
Share this drive publicly.
Upload a simple HTML file. Add any additional files (subfolders ok)
Open and "preview" the HTML file in Google Drive
Get the URL address for this folder
Create a direct link URL from your URL folder base
This URL should allow direct downloads of your large files.
[edit]
I forgot to add. If you use subfolders to organize your files, you simple use the folder name as you would expect in a URL hierarchy.
https://googledrive.com/host/<your public folders id string>/images/my-image.png
What I was looking to do
I created a custom Debian image with Virtual Box for Vagrant. I wanted to share this ".box" file with colleagues so they could put the direct link into their Vagrantfile.
In the end, I needed a direct link to the actual file.
Google Drive problem
If you set the file permissions to be publicly available and create/generate a direct access link by using something like the gdocs2direct tool or just crafting the link yourself:
https://docs.google.com/uc?export=download&id=<your file id>
You will get a cookie based verification code and prompt "Google could not scan this file" prompt, which won't work for things such as wget or Vagrantfile configs.
The code that it generates is a simple code that appends GET query variable ...&confirm=### to the string, but it's per user specific, so it's not like you can copy/paste that query variable for others.
But if you use the above "Web page hosting" method, you can get around that prompt.
I hope that helps!
If you face the "This file cannot be checked for viruses" intermezzo page, the download is not that easy.
You essentially need to first download the normal download link, which however redirects you to the "Download anyway" page. You need to store cookies from this first request, find out the link pointed to by the "Download anyway" button, and then use this link to download the file, but reusing the cookies you got from the first request.
Here's a bash variant of the download process using CURL:
curl -c /tmp/cookies "https://drive.google.com/uc?export=download&id=DOCUMENT_ID" > /tmp/intermezzo.html
curl -L -b /tmp/cookies "https://drive.google.com$(cat /tmp/intermezzo.html | grep -Po 'uc-download-link" [^>]* href="\K[^"]*' | sed 's/\&/\&/g')" > FINAL_DOWNLOADED_FILENAME
Notes:
this procedure will probably stop working after some Google changes
the grep command uses Perl syntax (-P) and the \K "operator" which essentially means "do not include anything preceding \K to the matched result. I don't know which version of grep introduced these options, but ancient or non-Ubuntu versions probably don't have it
a Java solution would be more or less the same, just take a HTTPS library which can handle cookies, and some nice text-parsing library
I know this is an old question but I could not find a solution to this problem after some research, so I am sharing what worked for me.
I have written this C# code for one of my projects. It can bypass the scan virus warning programmatically. The code can probably be converted to Java.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.IO;
using System.Net;
using System.Text;
public class FileDownloader : IDisposable
{
private const string GOOGLE_DRIVE_DOMAIN = "drive.google.com";
private const string GOOGLE_DRIVE_DOMAIN2 = "https://drive.google.com";
// In the worst case, it is necessary to send 3 download requests to the Drive address
// 1. an NID cookie is returned instead of a download_warning cookie
// 2. download_warning cookie returned
// 3. the actual file is downloaded
private const int GOOGLE_DRIVE_MAX_DOWNLOAD_ATTEMPT = 3;
public delegate void DownloadProgressChangedEventHandler( object sender, DownloadProgress progress );
// Custom download progress reporting (needed for Google Drive)
public class DownloadProgress
{
public long BytesReceived, TotalBytesToReceive;
public object UserState;
public int ProgressPercentage
{
get
{
if( TotalBytesToReceive > 0L )
return (int) ( ( (double) BytesReceived / TotalBytesToReceive ) * 100 );
return 0;
}
}
}
// Web client that preserves cookies (needed for Google Drive)
private class CookieAwareWebClient : WebClient
{
private class CookieContainer
{
private readonly Dictionary<string, string> cookies = new Dictionary<string, string>();
public string this[Uri address]
{
get
{
string cookie;
if( cookies.TryGetValue( address.Host, out cookie ) )
return cookie;
return null;
}
set
{
cookies[address.Host] = value;
}
}
}
private readonly CookieContainer cookies = new CookieContainer();
public DownloadProgress ContentRangeTarget;
protected override WebRequest GetWebRequest( Uri address )
{
WebRequest request = base.GetWebRequest( address );
if( request is HttpWebRequest )
{
string cookie = cookies[address];
if( cookie != null )
( (HttpWebRequest) request ).Headers.Set( "cookie", cookie );
if( ContentRangeTarget != null )
( (HttpWebRequest) request ).AddRange( 0 );
}
return request;
}
protected override WebResponse GetWebResponse( WebRequest request, IAsyncResult result )
{
return ProcessResponse( base.GetWebResponse( request, result ) );
}
protected override WebResponse GetWebResponse( WebRequest request )
{
return ProcessResponse( base.GetWebResponse( request ) );
}
private WebResponse ProcessResponse( WebResponse response )
{
string[] cookies = response.Headers.GetValues( "Set-Cookie" );
if( cookies != null && cookies.Length > 0 )
{
int length = 0;
for( int i = 0; i < cookies.Length; i++ )
length += cookies[i].Length;
StringBuilder cookie = new StringBuilder( length );
for( int i = 0; i < cookies.Length; i++ )
cookie.Append( cookies[i] );
this.cookies[response.ResponseUri] = cookie.ToString();
}
if( ContentRangeTarget != null )
{
string[] rangeLengthHeader = response.Headers.GetValues( "Content-Range" );
if( rangeLengthHeader != null && rangeLengthHeader.Length > 0 )
{
int splitIndex = rangeLengthHeader[0].LastIndexOf( '/' );
if( splitIndex >= 0 && splitIndex < rangeLengthHeader[0].Length - 1 )
{
long length;
if( long.TryParse( rangeLengthHeader[0].Substring( splitIndex + 1 ), out length ) )
ContentRangeTarget.TotalBytesToReceive = length;
}
}
}
return response;
}
}
private readonly CookieAwareWebClient webClient;
private readonly DownloadProgress downloadProgress;
private Uri downloadAddress;
private string downloadPath;
private bool asyncDownload;
private object userToken;
private bool downloadingDriveFile;
private int driveDownloadAttempt;
public event DownloadProgressChangedEventHandler DownloadProgressChanged;
public event AsyncCompletedEventHandler DownloadFileCompleted;
public FileDownloader()
{
webClient = new CookieAwareWebClient();
webClient.DownloadProgressChanged += DownloadProgressChangedCallback;
webClient.DownloadFileCompleted += DownloadFileCompletedCallback;
downloadProgress = new DownloadProgress();
}
public void DownloadFile( string address, string fileName )
{
DownloadFile( address, fileName, false, null );
}
public void DownloadFileAsync( string address, string fileName, object userToken = null )
{
DownloadFile( address, fileName, true, userToken );
}
private void DownloadFile( string address, string fileName, bool asyncDownload, object userToken )
{
downloadingDriveFile = address.StartsWith( GOOGLE_DRIVE_DOMAIN ) || address.StartsWith( GOOGLE_DRIVE_DOMAIN2 );
if( downloadingDriveFile )
{
address = GetGoogleDriveDownloadAddress( address );
driveDownloadAttempt = 1;
webClient.ContentRangeTarget = downloadProgress;
}
else
webClient.ContentRangeTarget = null;
downloadAddress = new Uri( address );
downloadPath = fileName;
downloadProgress.TotalBytesToReceive = -1L;
downloadProgress.UserState = userToken;
this.asyncDownload = asyncDownload;
this.userToken = userToken;
DownloadFileInternal();
}
private void DownloadFileInternal()
{
if( !asyncDownload )
{
webClient.DownloadFile( downloadAddress, downloadPath );
// This callback isn't triggered for synchronous downloads, manually trigger it
DownloadFileCompletedCallback( webClient, new AsyncCompletedEventArgs( null, false, null ) );
}
else if( userToken == null )
webClient.DownloadFileAsync( downloadAddress, downloadPath );
else
webClient.DownloadFileAsync( downloadAddress, downloadPath, userToken );
}
private void DownloadProgressChangedCallback( object sender, DownloadProgressChangedEventArgs e )
{
if( DownloadProgressChanged != null )
{
downloadProgress.BytesReceived = e.BytesReceived;
if( e.TotalBytesToReceive > 0L )
downloadProgress.TotalBytesToReceive = e.TotalBytesToReceive;
DownloadProgressChanged( this, downloadProgress );
}
}
private void DownloadFileCompletedCallback( object sender, AsyncCompletedEventArgs e )
{
if( !downloadingDriveFile )
{
if( DownloadFileCompleted != null )
DownloadFileCompleted( this, e );
}
else
{
if( driveDownloadAttempt < GOOGLE_DRIVE_MAX_DOWNLOAD_ATTEMPT && !ProcessDriveDownload() )
{
// Try downloading the Drive file again
driveDownloadAttempt++;
DownloadFileInternal();
}
else if( DownloadFileCompleted != null )
DownloadFileCompleted( this, e );
}
}
// Downloading large files from Google Drive prompts a warning screen and requires manual confirmation
// Consider that case and try to confirm the download automatically if warning prompt occurs
// Returns true, if no more download requests are necessary
private bool ProcessDriveDownload()
{
FileInfo downloadedFile = new FileInfo( downloadPath );
if( downloadedFile == null )
return true;
// Confirmation page is around 50KB, shouldn't be larger than 60KB
if( downloadedFile.Length > 60000L )
return true;
// Downloaded file might be the confirmation page, check it
string content;
using( var reader = downloadedFile.OpenText() )
{
// Confirmation page starts with <!DOCTYPE html>, which can be preceeded by a newline
char[] header = new char[20];
int readCount = reader.ReadBlock( header, 0, 20 );
if( readCount < 20 || !( new string( header ).Contains( "<!DOCTYPE html>" ) ) )
return true;
content = reader.ReadToEnd();
}
int linkIndex = content.LastIndexOf( "href=\"/uc?" );
if( linkIndex < 0 )
return true;
linkIndex += 6;
int linkEnd = content.IndexOf( '"', linkIndex );
if( linkEnd < 0 )
return true;
downloadAddress = new Uri( "https://drive.google.com" + content.Substring( linkIndex, linkEnd - linkIndex ).Replace( "&", "&" ) );
return false;
}
// Handles the following formats (links can be preceeded by https://):
// - drive.google.com/open?id=FILEID
// - drive.google.com/file/d/FILEID/view?usp=sharing
// - drive.google.com/uc?id=FILEID&export=download
private string GetGoogleDriveDownloadAddress( string address )
{
int index = address.IndexOf( "id=" );
int closingIndex;
if( index > 0 )
{
index += 3;
closingIndex = address.IndexOf( '&', index );
if( closingIndex < 0 )
closingIndex = address.Length;
}
else
{
index = address.IndexOf( "file/d/" );
if( index < 0 ) // address is not in any of the supported forms
return string.Empty;
index += 7;
closingIndex = address.IndexOf( '/', index );
if( closingIndex < 0 )
{
closingIndex = address.IndexOf( '?', index );
if( closingIndex < 0 )
closingIndex = address.Length;
}
}
return string.Concat( "https://drive.google.com/uc?id=", address.Substring( index, closingIndex - index ), "&export=download" );
}
public void Dispose()
{
webClient.Dispose();
}
}
And here's how you can use it:
// NOTE: FileDownloader is IDisposable!
FileDownloader fileDownloader = new FileDownloader();
// This callback is triggered for DownloadFileAsync only
fileDownloader.DownloadProgressChanged += ( sender, e ) => Console.WriteLine( "Progress changed " + e.BytesReceived + " " + e.TotalBytesToReceive );
// This callback is triggered for both DownloadFile and DownloadFileAsync
fileDownloader.DownloadFileCompleted += ( sender, e ) => Console.WriteLine( "Download completed" );
fileDownloader.DownloadFileAsync( "https://INSERT_DOWNLOAD_LINK_HERE", #"C:\downloadedFile.txt" );
#Case 1: download file with small size.
You can use url with format https://drive.google.com/uc?export=download&id=FILE_ID and then inputstream of file can be obtained directly.
#Case 2: download file with large size.
You stuck a wall of a virus scan alert page returned. By parsing html dom element, I tried to get link with confirm code under button "Download anyway" but it didn't work. Its may required cookie or session info.
enter image description here
SOLUTION:
Finally I found solution for two above cases. Just need to put httpConnection.setDoOutput(true) in connection step to get a Json.
)]}' { "disposition":"SCAN_CLEAN",
"downloadUrl":"http:www...",
"fileName":"exam_list_json.txt", "scanResult":"OK", "sizeBytes":2392}
Then, you can use any Json parser to read downloadUrl, fileName and sizeBytes.
You can refer follow snippet, hope it help.
private InputStream gConnect(String remoteFile) throws IOException{
URL url = new URL(remoteFile);
URLConnection connection = url.openConnection();
if(connection instanceof HttpURLConnection){
HttpURLConnection httpConnection = (HttpURLConnection) connection;
connection.setAllowUserInteraction(false);
httpConnection.setInstanceFollowRedirects(true);
httpConnection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows 2000)");
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("GET");
httpConnection.connect();
int reqCode = httpConnection.getResponseCode();
if(reqCode == HttpURLConnection.HTTP_OK){
InputStream is = httpConnection.getInputStream();
Map<String, List<String>> map = httpConnection.getHeaderFields();
List<String> values = map.get("content-type");
if(values != null && !values.isEmpty()){
String type = values.get(0);
if(type.contains("text/html")){
String cookie = httpConnection.getHeaderField("Set-Cookie");
String temp = Constants.getPath(mContext, Constants.PATH_TEMP) + "/temp.html";
if(saveGHtmlFile(is, temp)){
String href = getRealUrl(temp);
if(href != null){
return parseUrl(href, cookie);
}
}
} else if(type.contains("application/json")){
String temp = Constants.getPath(mContext, Constants.PATH_TEMP) + "/temp.txt";
if(saveGJsonFile(is, temp)){
FileDataSet data = JsonReaderHelper.readFileDataset(new File(temp));
if(data.getPath() != null){
return parseUrl(data.getPath());
}
}
}
}
return is;
}
}
return null;
}
And
public static FileDataSet readFileDataset(File file) throws IOException{
FileInputStream is = new FileInputStream(file);
JsonReader reader = new JsonReader(new InputStreamReader(is, "UTF-8"));
reader.beginObject();
FileDataSet rs = new FileDataSet();
while(reader.hasNext()){
String name = reader.nextName();
if(name.equals("downloadUrl")){
rs.setPath(reader.nextString());
} else if(name.equals("fileName")){
rs.setName(reader.nextString());
} else if(name.equals("sizeBytes")){
rs.setSize(reader.nextLong());
} else {
reader.skipValue();
}
}
reader.endObject();
return rs;
}
This seems to be updated again as of May 19, 2015:
How I got it to work:
As in jmbertucci's recently updated answer, make your folder public to everyone. This is a bit more complicated than before, you have to click Advanced to change the folder to "On - Public on the web."
Find your folder UUID as before--just go into the folder and find your UUID in the address bar:
https://drive.google.com/drive/folders/<folder UUID>
Then head to
https://googledrive.com/host/<folder UUID>
It will redirect you to an index type page with a giant subdomain, but you should be able to see the files in your folder. Then you can right click to save the link to the file you want (I noticed that this direct link also has this big subdomain for googledrive.com). Worked great for me with wget.
This also seems to work with others' shared folders.
e.g.,
https://drive.google.com/folderview?id=0B7l10Bj_LprhQnpSRkpGMGV2eE0&usp=sharing
maps to
https://googledrive.com/host/0B7l10Bj_LprhQnpSRkpGMGV2eE0
And a right click can save a direct link to any of those files.
Using a Service Account might work for you.
Check this out:
wget https://raw.githubusercontent.com/circulosmeos/gdown.pl/master/gdown.pl
chmod +x gdown.pl
./gdown.pl https://drive.google.com/file/d/FILE_ID/view TARGET_PATH
Update as of August 2020:
This is what worked for me recently -
Upload your file and get a shareable link which anyone can see(Change permission from "Restricted" to "Anyone with the Link" in the share link options)
Then run:
SHAREABLE_LINK=<google drive shareable link>
curl -L https://drive.google.com/uc\?id\=$(echo $SHAREABLE_LINK | cut -f6 -d"/")
If you just want to programmatically (as oppossed to giving the user a link to open in a browser) download a file through the Google Drive API, I would suggest using the downloadUrl of the file instead of the webContentLink, as documented here: https://developers.google.com/drive/web/manage-downloads
https://github.com/google/skicka
I used this command line tool to download files from Google Drive. Just follow the instructions in Getting Started section and you should download files from Google Drive in minutes.
For any shared link replace FILENAME and FILEID, (for very large files requiring confirmation):
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.confirm=([0-9A-Za-z_]+)./\1\n/p')&id=FILEID" -O FILENAME && rm -rf /tmp/cookies.txt
(For small files):
wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O FILENAME
I would consider downloading from the link, scraping the page that you get to grab the confirmation link, and then downloading that.
If you look at the "download anyway" URL it has an extra confirm query parameter with a seemingly randomly generated token. Since it's random...and you probably don't want to figure out how to generate it yourself, scraping might be the easiest way without knowing anything about how the site works.
You may need to consider various scenarios.
I simply create a javascript so that it automatically capture the link and download and close the tab with the help of tampermonkey.
// ==UserScript==
// #name Bypass Google drive virus scan
// #namespace SmartManoj
// #version 0.1
// #description Quickly get the download link
// #author SmartManoj
// #match https://drive.google.com/uc?id=*&export=download*
// #grant none
// ==/UserScript==
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function demo() {
await sleep(5000);
window.close();
}
(function() {
location.replace(document.getElementById("uc-download-link").href);
demo();
})();
Similarly you can get the html source of the url and download in java.
I faced an issue in direct download because I was logged in using multiple Google accounts.
Solution is append authUser=0 parameter. Sample request URL to download :https://drive.google.com/uc?id=FILEID&authuser=0&export=download
https://drive.google.com/uc?export=download&id=FILE_ID replace the FILE_ID with file id.
if you don't know were is file id then check this article Article LINK
I'm looking for a way to retrieve the Tomcat log within a webapp. In the past I've seen this feature provided in other webapps, usually dumping the log in a Servlet.
I'm using slf4j (with log4j) and Tomcat 6. I haven't found anything relevant in the Tomcat docs, although the JMX API looks like it might provide something useful? I'm not too concerned whether the output represents just the webapp logging or the entire Tomcat log, either will suffice.
Ideally, I'm hoping for a solution that does not involve scraping the log from the filesystem, although if that is the only way, it would be great if the log directory could be calculated at runtime...
Scraping the log from the filesystem is probably the easiest way to go. You can get the log directly programatically using System.getProperty("catalina.base") + "/logs".
Otherwise you could set up an additional appender in your log4j configuration to log to something like JDBC, JMS, a Writer, etc. Whatever makes sense for your app.
This function will get the most recent log file matching a given prefix. You do not need to know what directory the logs are written to.
public static File locateLogFile( final String prefixToMatch ) {
File result = null;
Handler[] handlers = LogManager.getLogManager().getLogger( "" ).getHandlers();
try {
for( Handler handler : handlers ) {
Field directoryField;
Field prefixField;
try {
//These are private fields in the juli FileHandler class
directoryField = handler.getClass().getDeclaredField( "directory" );
prefixField = handler.getClass().getDeclaredField( "prefix" );
directoryField.setAccessible( true );
prefixField.setAccessible( true );
} catch( NoSuchFieldException e ) {
continue;
}
String directory = (String)directoryField.get( handler );
if( prefixToMatch.equals( prefixField.get( handler ) ) ) {
File logDirectory = new File( directory );
File[] logFiles = logDirectory.listFiles( new FileFilter() {
public boolean accept( File pathname ) {
return pathname.getName().startsWith( prefixToMatch );
}
} );
if( logFiles.length == 0 ) continue;
Arrays.sort( logFiles );
result = logFiles[ logFiles.length - 1 ];
break;
}
}
} catch( IllegalAccessException e ) {
log.log( Level.WARNING, "Couldn't get log file", e );
}
return result;
}
Is there any way to access the Windows Event Log from a java class. Has anyone written any APIs for this, and would there be any way to access the data from a remote machine?
The scenario is:
I run a process on a remote machine, from a controlling Java process.
This remote process logs stuff to the Event Log, which I want to be able to see in the controlling process.
Thanks in advance.
http://www.j-interop.org/ is an open-source Java library that implements the DCOM protocol specification without using any native code. (i.e. you can use it to access DCOM objects on a remote Windows host from Java code running on a non-Windows client).
Microsoft exposes a plethora of system information via Windows Management Instrumentation (WMI). WMI is remotely accessible via DCOM, and considerable documentation on the subject exists on Microsoft's site. As it happens, you can access the Windows Event Logs via this remotely accessible interface.
By using j-interop you can create an instance of the WbemScripting.SWbemLocator WMI object remotely, then connect to Windows Management Instrumentation (WMI) services on the remote Windows host. From there you can submit a query that will inform you whenever a new event log entry is written.
Note that this does require that you have DCOM properly enabled and configured on the remote Windows host, and that appropriate exceptions have been set up in any firewalls. Details on this can be searched online, and are also referenced from the j-interop site, above.
The following example connects to a remote host using its NT domain, hostname, a username and a password, and sits in a loop, dumping every event log entry as they are logged by windows. The user must have been granted appropriate remote DCOM access permissions, but does not have to be an administrator.
import java.io.IOException;
import java.util.logging.Level;
import org.jinterop.dcom.common.JIException;
import org.jinterop.dcom.common.JISystem;
import org.jinterop.dcom.core.JIComServer;
import org.jinterop.dcom.core.JIProgId;
import org.jinterop.dcom.core.JISession;
import org.jinterop.dcom.core.JIString;
import org.jinterop.dcom.core.JIVariant;
import org.jinterop.dcom.impls.JIObjectFactory;
import org.jinterop.dcom.impls.automation.IJIDispatch;
public class EventLogListener
{
private static final String WMI_DEFAULT_NAMESPACE = "ROOT\\CIMV2";
private static JISession configAndConnectDCom( String domain, String user, String pass ) throws Exception
{
JISystem.getLogger().setLevel( Level.OFF );
try
{
JISystem.setInBuiltLogHandler( false );
}
catch ( IOException ignored )
{
;
}
JISystem.setAutoRegisteration( true );
JISession dcomSession = JISession.createSession( domain, user, pass );
dcomSession.useSessionSecurity( true );
return dcomSession;
}
private static IJIDispatch getWmiLocator( String host, JISession dcomSession ) throws Exception
{
JIComServer wbemLocatorComObj = new JIComServer( JIProgId.valueOf( "WbemScripting.SWbemLocator" ), host, dcomSession );
return (IJIDispatch) JIObjectFactory.narrowObject( wbemLocatorComObj.createInstance().queryInterface( IJIDispatch.IID ) );
}
private static IJIDispatch toIDispatch( JIVariant comObjectAsVariant ) throws JIException
{
return (IJIDispatch) JIObjectFactory.narrowObject( comObjectAsVariant.getObjectAsComObject() );
}
public static void main( String[] args )
{
if ( args.length != 4 )
{
System.out.println( "Usage: " + EventLogListener.class.getSimpleName() + " domain host username password" );
return;
}
String domain = args[ 0 ];
String host = args[ 1 ];
String user = args[ 2 ];
String pass = args[ 3 ];
JISession dcomSession = null;
try
{
// Connect to DCOM on the remote system, and create an instance of the WbemScripting.SWbemLocator object to talk to WMI.
dcomSession = configAndConnectDCom( domain, user, pass );
IJIDispatch wbemLocator = getWmiLocator( host, dcomSession );
// Invoke the "ConnectServer" method on the SWbemLocator object via it's IDispatch COM pointer. We will connect to
// the default ROOT\CIMV2 namespace. This will result in us having a reference to a "SWbemServices" object.
JIVariant results[] =
wbemLocator.callMethodA( "ConnectServer", new Object[] { new JIString( host ), new JIString( WMI_DEFAULT_NAMESPACE ),
JIVariant.OPTIONAL_PARAM(), JIVariant.OPTIONAL_PARAM(), JIVariant.OPTIONAL_PARAM(), JIVariant.OPTIONAL_PARAM(), new Integer( 0 ),
JIVariant.OPTIONAL_PARAM() } );
IJIDispatch wbemServices = toIDispatch( results[ 0 ] );
// Now that we have a SWbemServices DCOM object reference, we prepare a WMI Query Language (WQL) request to be informed whenever a
// new instance of the "Win32_NTLogEvent" WMI class is created on the remote host. This is submitted to the remote host via the
// "ExecNotificationQuery" method on SWbemServices. This gives us all events as they come in. Refer to WQL documentation to
// learn how to restrict the query if you want a narrower focus.
final String QUERY_FOR_ALL_LOG_EVENTS = "SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA 'Win32_NTLogEvent'";
final int RETURN_IMMEDIATE = 16;
final int FORWARD_ONLY = 32;
JIVariant[] eventSourceSet =
wbemServices.callMethodA( "ExecNotificationQuery", new Object[] { new JIString( QUERY_FOR_ALL_LOG_EVENTS ), new JIString( "WQL" ),
new JIVariant( new Integer( RETURN_IMMEDIATE + FORWARD_ONLY ) ) } );
IJIDispatch wbemEventSource = (IJIDispatch) JIObjectFactory.narrowObject( ( eventSourceSet[ 0 ] ).getObjectAsComObject() );
// The result of the query is a SWbemEventSource object. This object exposes a method that we can call in a loop to retrieve the
// next Windows Event Log entry whenever it is created. This "NextEvent" operation will block until we are given an event.
// Note that you can specify timeouts, see the Microsoft documentation for more details.
while ( true )
{
// this blocks until an event log entry appears.
JIVariant eventAsVariant = (JIVariant) ( wbemEventSource.callMethodA( "NextEvent", new Object[] { JIVariant.OPTIONAL_PARAM() } ) )[ 0 ];
IJIDispatch wbemEvent = toIDispatch( eventAsVariant );
// WMI gives us events as SWbemObject instances (a base class of any WMI object). We know in our case we asked for a specific object
// type, so we will go ahead and invoke methods supported by that Win32_NTLogEvent class via the wbemEvent IDispatch pointer.
// In this case, we simply call the "GetObjectText_" method that returns us the entire object as a CIM formatted string. We could,
// however, ask the object for its property values via wbemEvent.get("PropertyName"). See the j-interop documentation and examples
// for how to query COM properties.
JIVariant objTextAsVariant = (JIVariant) ( wbemEvent.callMethodA( "GetObjectText_", new Object[] { new Integer( 1 ) } ) )[ 0 ];
String asText = objTextAsVariant.getObjectAsString().getString();
System.out.println( asText );
}
}
catch ( Exception e )
{
e.printStackTrace();
}
finally
{
if ( null != dcomSession )
{
try
{
JISession.destroySession( dcomSession );
}
catch ( Exception ex )
{
ex.printStackTrace();
}
}
}
}
}
~
On the Java side, you'll need a library that allows you to make native calls. Sun offers JNI, but it sounds like sort of a pain. Also consider:
https://github.com/twall/jna/
http://johannburkard.de/software/nativecall/
http://www.jinvoke.com/
On the Windows side, the function you're after is OpenEventLog. This should allow you to access a remote event log. See also Querying for Event Information.
If that doesn't sound right, I also found this for parsing the log files directly (not an approach I'd recommend but interesting nonetheless):
http://msdn.microsoft.com/en-us/library/bb309026.aspx
http://objectmix.com/java/75154-regarding-windows-event-log-file-parser-java.html
Read this article.
JNA 3.2.8 has both methods to read and write from the Windows event log.
You can see an example of write in log4jna.
Here's an example of read:
EventLogIterator iter = new EventLogIterator("Application");
while(iter.hasNext()) {
EventLogRecord record = iter.next();
System.out.println(record.getRecordNumber()
+ ": Event ID: " + record.getEventId()
+ ", Event Type: " + record.getType()
+ ", Event Source: " + record.getSource());
}
If you want true event log access from a remote machine, you will have to find a library which implements the EventLog Remoting Protocol Specification. Unfortunately, I have not yet found any such library in Java. However, much of the foundation for implementing this protocol has already been laid by the JCIFS and JARAPAC projects. The protocol itself (if I'm not mistaken) runs on top of the DCE/RPC protocol (implemented by JARAPAC) which itself runs on top of the SMB protocol (implemented by JCIFS).
I have already been using JCIFS and JARAPAC to implement some of EventLog's cousin protocols, such as remote registry access. I may be blind, but documentation seemed a little scarce regarding JARAPAC. If you are interested in implementing this, I can share with you what I have learned when I get some spare time!
Later!
there are a million (and one) options here ;)
you could look at sigar
http://cpansearch.perl.org/src/DOUGM/hyperic-sigar-1.6.3-src/docs/javadoc/org/hyperic/sigar/win32/EventLog.html
mind the licensing though....
or you could be quick and dirty and just periodically execute (and capture the output)
D:>cscript.exe c:\WINDOWS\system32\eventquery.vbs /v
then use event filtering params to refine the results etc...
http://technet.microsoft.com/en-us/library/cc772995(WS.10).aspx