We Keep Coding

HtmlUnit and HTTPS pages

I'm trying to make a program that checks avaliable positions and books the first avaliable one. I started writing it and i ran into a problem pretty early.
The problem is that when I try to connect with the site (which is https) the program doesn't do anything. It doesn't throw an error, it doesn't crash. And the weirdest thing is that it works with some https websites and with some it doesn't.
I've spent countless hours trying to resolve this problem. I tried using htmlunitdriver and it still doesn't work. Please help.
private final WebClient webc = new WebClient(BrowserVersion.CHROME);
webc.getCookieManager().setCookiesEnabled(true);
HtmlPage loginpage = webc.getPage(loginurl);
System.out.println(loginpage.getTitleText());
I'm getting really frustrated with this. Thank you in advance.

As far as i can see this has nothing to do with HttpS. It is a good idea to do some traffic analysis using Charles or Fiddler.
What you can see....
The page returned from the server as response to your first call to https://online.enel.pl/ loads some external javascript. And then the story begins:
This JS looks like
(function() {
var z = "";
var b = "766172205f3078666.....";
eval((function() {
for (var i = 0; i < b.length; i += 2) {
z += String.fromCharCode(parseInt(b.substring(i, i + 2), 16));
}
return z;
})());
})();
As you can see someone likes to hide the real javascript that gets processed.
Next step is to check the javascript after this simple decoding
It is really huge and looks like this
var _0xfbfd = ['\x77\x71\x30\x6b\x77 ....
(function (_0x2ea96d, _0x460da4) {
var _0x1da805 = function (_0x55e996) {
while (--_0x55e996) {
_0x2ea96d['\x70\x75\x73\x68'](_0x2ea96d['\x73\x68\x69\x66\x74']());
}
};
.....
Ok now we have obfuscated javascript. If you like you can start with http://ddecode.com/hexdecoder/ to get some more readable text but this was the step where i have stopped my analysis. Looks like this script does some really bad things or someone still believes in security by obscurity.
If you run this with HtmlUnit, this codes gets interpreted - yes the decoding works and the code runs. Sadly this code runs endless (maybe because of an error or some incompatibility with real browsers).
If you like to get this working, you have to figure out, where the error is and open an bug report for HtmlUnit. For this you can simply start with a small local HtmlFile and include the code from the first external javascript. Then add some log statements to get the decoded version. Then replace this with the decoded version and try to understand what is going on. You can start adding alert statements and check if the code in HtmlUnit follows the same path as browsers do. Sorry but my time is to limited to do all this work but i really like to help/fix if you can point to a specific function in HtmlUnit that works different from real browsers.

Without the URL that you are querying it is dificult to say what could be wrong. However, having worked with HTML unit some time back I found that it was failing with many sites that I needed to get data from. The site owners will do many things to avoid you using programs to access them and you might have to resort to using some lower level library like Apache HTTP components where you have more control over what is going on under the hood.
Also check if the website is constructed using JavaScript which is getting more and more popular but making it increasingly dificult to use programs to interrogate the content.

In Java, starting a test in selenium with the DefaultSelenium object how do I find which browser the test is running on?

Consider a simple DefaultSelenium object
DefaultSelenium sel = new
DefaultSelenium("http://localhost:8080/myapp",4444,"*iexplore","/myAppLevel1");
Now my server is set with the option of -forcedBrowserMode "*firefox" in the command line when I start it up. However, I have 2 different batch files to start the Server, one forced in firefox, one forced to IE. FYI, the -forcedBrowserMode overrides the settings within of the instantiated java object.
The problem is from java, I can't seem to find a way to determine which browser my DefaultSelenium object is running on... I was thinking something like:
sel.getBrowserName();
But nothing like that exists. Are there any other creative ways of doing this?
I need to know because with a GWT web application, to click on a button you need to do it differently based on the browser. As well, you may wonder why I even use the -forcedBrowserMode, because then I can use custom setup firefox/ie installs to test on.
Thanks in advance for help!

I think that you can get the browser by executing some JavaScript, for example verify navigator.userAgent or any browser specific object, for example document.defaultView will be null in IE and not null in FF, something like this:
DefaultSelenium sel = ...
String res = sel.getEval("document.defaultView ? false : true");
boolean isIE = "true".equals(res);

Problem loading R own created libraries in Java/JRI code

I created my own new R library (called "Media"). There is no problem when I try to load it with RGui, and I can call the functions defined in the new package. This is how I load it:
> library(Media)
But, I'm also trying to call that functions from Java/JRI code, and when I load the new R package, Java doesn't seem to find the pacakge, throwing the message "Error in library(Media) : object 'Media' not found"
This is my current code using JRI:
REXP rexpSetFolder = re.eval("setwd('C:/Users/Albert/Documents')");
REXP rexpFolder = re.eval("getwd()");
System.out.println(rexpFolder.asString());
REXP rexpLoad = re.eval("library(Media)"); // fails
It also fails without the 'setwd' command, and simple calls to existing R functions work fine. I'm using R 2.10 and the latest JRI 0.5-0 under Windows.
Any help would be appreciated.
Thank you very much.
Edit:
The parameter lib.loc seems to work, at least this sentence does not return an error:
library("Media", lib.loc = "c:/Users/Albert/Documents")
But after that, calling a function in the package with re.eval("myfunction()"); still fails, as the function is not properly found.

You can modify the library path - see ?.libPaths in R, you simply want to add your private library to the path. The GUI does that for you, but if you are outside it doesn't happen. For example:
re.eval(".libPaths('c:/users/foo/Documents/R')");
Then load your package.

Did you install the library properly first? You might want to try using the lib.loc parameter.
library("Media", lib.loc = "c:/Users/Albert/Documents")

My work-around was to copy the package from my personal library (%USERPROFILE%\Documents\R) to the global library (%R_HOME%\library).
It's not the best because this requires Administrator privileges which not all users will have...

How to set/get profile data with XMPP using Smack

I am working on a XMPP client on Android, using the Smack library. The roster/messaging/presence stuff is running very well. However, I didn't find a way to store additional profile information (userpicture, the dogs name, ...).
The only way I see from googling is using VCards. But it simply did not work. I tried the following:
VCard vCard = new VCard();
vCard.load(connection);
vCard.setEmailHome("meine_home#email.de");
vCard.setLastName("Scheller");
vCard.setField("blafasel", "asdf");
vCard.save(connection);
Then I was looking for a way to see that VCard information. It did neither show up in iChat nor in this System.out:
vCard.load(connection, user);
System.out.println(user + " has this vCard: " + vCard.toXML());
So anything went wrong, but theres no indication what it was. I tried this with the google talk server and my own copy of openfire with the same result. Btw, I am using this version of Smack: http://davanum.wordpress.com/2007/12/31/android-just-use-smack-api-for-xmpp/
What am I doing wrong here? What is the correct way of storing profile related information with Smack?

I have checked out the source of Smack and went through the important parts with a debugger, as well as using the Smack Debug Window. The problem is inside the VCard implementation of the Smack API. Saving a VCard does work as described, however the loading is broken.
parseIQ(XmlPullParser parser) is part of the PacketReader.java class and handles different types of packages. It only handles tags with the following namespaces:
"jabber:iq:auth",
"jabber:iq:roster",
"jabber:iq:register",
"urn:ietf:params:xml:ns:xmpp-bind"
It also looks if there is any registered IQProvider in the ProviderManager. And this is the root of my problem. There is no IQProvider for VCards registered. So whatever information is inside of the vCard tag simply gets dropped.
It is not too hard to register this IQProvider though:
ProviderManager.getInstance().addIQProvider("vCard", "vcard-temp", new VCardProvider());
This solved my little example above for saving my own vCard and downloading it again. I am still having trouble with downloading other users vcards... Gonna have a closer look into this and maybe open up another thread for that issue.

You can use the following code to get info.
VCard card = new VCard();
card.load(connection, "user#fqdn");
System.out.println("Voice: "+card.getPhoneHome("VOICE"));

Try setting a vCard for that user with another client first, and see how that changes your results. In order to diagnose further, you'll need to turn on protocol debugging in Smack (use "-Dsmack.debugEnabled=true" on a desktop machine), and post the relevant bits here.

Pinning a Java application to the Windows 7 taskbar

I use Launch4j as a wrapper for my Java application under Windows 7, which, to my understanding, in essence forks an instance of javaw.exe that in turn interprets the Java code. As a result, when attempting to pin my application to the task bar, Windows instead pins javaw.exe. Without the required command line, my application will then not run.
As you can see, Windows also does not realize that Java is the host application: the application itself is described as "Java(TM) Platform SE binary".
I have tried altering the registry key HKEY_CLASSES_ROOT\Applications\javaw.exe to add the value IsHostApp. This alters the behavior by disabling pinning of my application altogether; clearly not what I want.
After reading about how Windows interprets instances of a single application (and a phenomenon discussed in this question), I became interested in embedding a Application User Model ID (AppUserModelID) into my Java application.
I believe that I can resolve this by passing a unique AppUserModelID to Windows. There is a shell32 method for this, SetCurrentProcessExplicitAppUserModelID. Following Gregory Pakosz suggestion, I implemented it in an attempt to have my application recognized as a separate instance of javaw.exe:
NativeLibrary lib;
try {
lib = NativeLibrary.getInstance("shell32");
} catch (Error e) {
Logger.out.error("Could not load Shell32 library.");
return;
}
Object[] args = { "Vendor.MyJavaApplication" };
String functionName = "SetCurrentProcessExplicitAppUserModelID";
try {
Function function = lib.getFunction(functionName);
int ret = function.invokeInt(args);
if (ret != 0) {
Logger.out.error(function.getName() + " returned error code "
+ ret + ".");
}
} catch (UnsatisfiedLinkError e) {
Logger.out.error(functionName + " was not found in "
+ lib.getFile().getName() + ".");
// Function not supported
}
This appears to have no effect, but the function returns without error. Diagnosing why is something of a mystery to me. Any suggestions?
Working implementation
The final implementation that worked is the answer to my follow-up question concerning how to pass the AppID using JNA.
I had awarded the bounty to Gregory Pakosz' brilliant answer for JNI that set me on the right track.
For reference, I believe using this technique opens the possibility of using any of the APIs discussed in this article in a Java application.

I don't have Windows 7 but here is something that might get you started:
On the Java side:
package com.stackoverflow.homework;
public class MyApplication
{
static native boolean setAppUserModelID();
static
{
System.loadLibrary("MyApplicationJNI");
setAppUserModelID();
}
}
And on the native side, in the source code of the `MyApplicationJNI.dll library:
JNIEXPORT jboolean JNICALL Java_com_stackoverflow_homework_MyApplication_setAppUserModelID(JNIEnv* env)
{
LPCWSTR id = L"com.stackoverflow.homework.MyApplication";
HRESULT hr = SetCurrentProcessExplicitAppUserModelID(id);
return hr == S_OK;
}
Your question explicitly asked for a JNI solution. However, since your application doesn't need any other native method, jna is another solution which will save you from writing native code just for the sake of forwarding to the windows api. If you decide to go jna, pay attention to the fact that SetCurrentProcessExplicitAppUserModelID() is expecting a UTF-16 string.
When it works in your sandbox, the next step is to add operating system detection in your application as SetCurrentProcessExplicitAppUserModelID() is obviously only available in Windows 7:
you may do that from the Java side by checking that System.getProperty("os.name"); returns "Windows 7".
if you build from the little JNI snippet I gave, you can enhance it by dynamically loading the shell32.dll library using LoadLibrary then getting back the SetCurrentProcessExplicitAppUserModelID function pointer using GetProcAddress. If GetProcAddress returns NULL, it means the symbol is not present in shell32 hence it's not Windows 7.
EDIT: JNA Solution.
References:
The JNI book for more JNI examples
Java Native Access (JNA)

There is a Java library providing the new Windows 7 features for Java. It's called J7Goodies by Strix Code. Applications using it can be properly pinned to the Windows 7 taskbar. You can also create your own jump lists, etc.

I have implemented access to the SetCurrentProcessExplicitAppUserModelID method using JNA and it works quite well when used as the MSDN documentation suggests. I've never used the JNA api in the way you have in your code snippet. My implementation follows the typical JNA usage instead.
First the Shell32 interface definition:
interface Shell32 extends StdCallLibrary {
int SetCurrentProcessExplicitAppUserModelID( WString appID );
}
Then using JNA to load Shell32 and call the function:
final Map<String, Object> WIN32API_OPTIONS = new HashMap<String, Object>() {
{
put(Library.OPTION_FUNCTION_MAPPER, W32APIFunctionMapper.UNICODE);
put(Library.OPTION_TYPE_MAPPER, W32APITypeMapper.UNICODE);
}
};
Shell32 shell32 = (Shell32) Native.loadLibrary("shell32", Shell32.class,
WIN32API_OPTIONS);
WString wAppId = new WString( "Vendor.MyJavaApplication" );
shell32.SetCurrentProcessExplicitAppUserModelID( wAppId );
Many of the API's in the last article you mentioned make use of Windows COM which is quite difficult to use directly with JNA. I have had some success creating a custom DLL to call these API's (eg. using the SHGetPropertyStoreForWindow to set a different app ID for a submodule window) which I then use JNA to access at runtime.

Try to use JSmooth. I use always this one. In JSmooth is there an option under Skeleton by Windowed Wrapper called
Lauch java app in exe process
See on this image.
(source: andrels.com)
Also command line arguments can be passed.
I think this can be a solution for you.
Martijn

SetCurrentProcessExplicitAppUserModelID (or SetAppID()) would in fact do what you're trying to do. However, it might be easier to modify your installer to set the AppUserModel.ID property on your shortcut - quoting from the Application User Model ID document mentioned above:
In the System.AppUserModel.ID property of the application's shortcut file. A shortcut (as an IShellLink, CLSID_ShellLink, or a .lnk file) supports properties through IPropertyStore and other property-setting mechanisms used throughout the Shell. This allows the taskbar to identify the proper shortcut to pin and ensures that windows belonging to the process are appropriately associated with that taskbar button.
Note: The System.AppUserModel.ID property should be applied to a shortcut when that shortcut is created. When using the Microsoft Windows Installer (MSI) to install the application, the MsiShortcutProperty table allows the AppUserModelID to be applied to the shortcut when it is created during installation.

The latest jna-platform library now includes JNA bindings for SetCurrentProcessExplicitAppUserModelID:
https://github.com/java-native-access/jna/pull/680

I fixed mine without any ID settings.
There is an option in Launch4J if you are using it and you say you do then...
You can change the header to JNI Gui and then wrap it around the jar with the JRE.
The good thing is that it runs .exe in the process now instead on running javaw.exe with your jar. It probably does it under the hood (not sure).
Also I have noticed also that it takes around 40-50% less CPU resource which is even better!
And the pinning works fine and all that window features are enabled.
I hope it helps to someone as I spent nearly 2 days trying to solve that issue with my undecorated javafx app.