I'm trying to get all entries on an LDAP server using Spring LDAP (version 2.3.2). Within my code, I make use of PagedResultsDirContextProcessor to paginate through all the result. This works fine on the servers which support PagedResultsControl.
However, I now need to connect to an LDAP server which does not support PagedResultsControl. How can I get all entries without using PagedResultsControl?
You can use VirtualListView via JNDI. You have to retrieve and re-supply the 'contextID' to paginate, as follows:
static final int LIST_SIZE = 20; // Adjust to suit
#Test
public void TestVLV() throws NamingException, IOException
{
Hashtable<String,Object> env = new Hashtable<>();
env.put(Context.INITIAL_CONTEXT_FACTORY,
"com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.PROVIDER_URL, "ldap://localhost");
env.put(Context.SECURITY_AUTHENTICATION, "simple");
env.put(Context.SECURITY_PRINCIPAL, "cn=XXXXXXX");
env.put(Context.SECURITY_CREDENTIALS, "YYYYYYY");
try
{
/* Create initial context with no connection request controls */
LdapContext ctx = new InitialLdapContext(env, null);
/* Sort Control is required for VLV to work */
SortKey[] sortKeys =
{
// sort by cn
new SortKey("cn", true, "caseIgnoreOrderingMatch")
};
// Note: the constructors for SortControl that take String or String[]
// as the first argument produce 'no ordering rule' errors with OpenLDAP.
SortControl sctl = new SortControl(
// "cn",
// new String[]{"cn"},
sortKeys,
Control.CRITICAL);
/* VLV that returns the first 20 answers */
VirtualListViewControl vctl =
new VirtualListViewControl(1, 0, 0, LIST_SIZE-1, Control.CRITICAL);
/* Set context's request controls */
ctx.setRequestControls(new Control[]
{
sctl,
vctl
});
int count = 0;
SearchControls sc = new SearchControls(SearchControls.SUBTREE_SCOPE, 0, 0, null, false, false);
for (;;)
{
/* Perform search */
// System.out.println("namespace="+ctx.getNameInNamespace());
// System.out.println("count limit="+sc.getCountLimit());
// System.out.println("search scope="+sc.getSearchScope());
NamingEnumeration<SearchResult> ne =
ctx.search("ou=Users,dc=xxxx,dc=com", "(objectClass={0})", new String[]{"inetOrgPerson"}, sc);
/* Enumerate search results */
while (ne.hasMore())
{
count++;
SearchResult sr = ne.next();
// System.out.println(i+": "+sr.getName());
System.out.println(count+": "+sr.getNameInNamespace());
}
ne.close();
// Get the contextID.
Control[] controls = ctx.getResponseControls();
VirtualListViewResponseControl vlvrc = null;
byte[] contextID = null;
for (int j = 0; j < controls.length; j++)
{
if (controls[j] instanceof VirtualListViewResponseControl)
{
vlvrc = (VirtualListViewResponseControl)controls[j];
contextID = vlvrc.getContextID();
System.out.println("contextID=0x"+new BigInteger(1,contextID).toString(16));
if (contextID != null)
{
vctl = new VirtualListViewControl(vlvrc.getTargetOffset()+LIST_SIZE, 0, 0, LIST_SIZE-1, Control.CRITICAL);
vctl.setContextID(contextID);
ctx.setRequestControls(new Control[]
{
sctl,
vctl
});
}
break; // there should only be one VLV response control, and we're not interested in anything else.
}
}
if (vlvrc != null && contextID != null && count < vlvrc.getListSize())
{
System.out.println("Continuing");
}
else
{
System.out.println("Finished");
break;
}
}
ctx.close();
}
finally
{
}
}
Adjust the authentication and search root and filter to suit yourself, of course.
And to test whether it is supported (although an 'unsupported critical control' exception from the above code will tell you just as well):
/**
* Is VLV Control supported?
*
* Query the rootDSE object to find out if VLV Control is supported.
* #return true if it is supported.
*/
static boolean isVLVControlSupported(LdapContext ctx)
throws NamingException
{
String[] returningAttributes =
{
"supportedControl"
};
// Fetch the supportedControl attribute of the rootDSE object.
Attributes attrs = ctx.getAttributes("", returningAttributes);
Attribute attr = attrs.get("supportedControl");
System.out.println("supportedControls="+attr);
if (attr != null)
{
// Fast way to check. add() would have been just as good. Does no damage to the DIT.
return attr.remove(VLV_CONTROL_OID);
}
return false;
}
The VirtualListViewControl and VirtualListViewResponseControl are part of the Sun/Oracle LDAP Booster Pack, which you can obtain via Maven as:
<dependency>
<groupId>com.sun</groupId>
<artifactId>ldapbp</artifactId>
<version>1.0</version>
<type>jar</type>
</dependency>
Super frustrating..
I wouldn't really recommend it, but you could pull off something like this where you manually paginate by cn/sn..
List<String> alphabetRange = getAlphabetRange();
for (int i = 0; i < alphabetRange.size() - 1; i++) {
String filter = "(&(sn>=" + alphabetRange.get(i) + ")" + "(sn<=" + alphabetRange.get(i + 1) + " ))";
NamingEnumeration<SearchResult> searchResult = context.search(base_dn, filter, controls);
while (searchResult.hasMore()) {
// searchResult.next().getAttributes() and do something with it
}
}
private List<String> getAlphabetRange() {
List<String> result = new ArrayList<>();
for (char alph = 'A'; alph <= 'Z'; alph++) {
if (alph == 'S') {
result.add("S");
result.add("Sd");
} else {
result.add(String.valueOf(alph));
}
}
result.add("Zz");
return result;
}
as soon as you run over (i.e. 1.000) in one "page", you will face the dreaded javax.naming.SizeLimitExceededException. You might add more "pages", like in that example between [S-Sd][Sd-T]
What exacly servers are you using?
if server has no queue limits you cant try set unlimited search:
SearchControls controls = new SearchControls();
controls.setTimeLimit(0);
controls.setCountLimit(0);
Related
I'm using the below code to fire a standard DNS query. I need the TTL from the response. However the output from JNDI does not contain the TTL
Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.dns.DnsContextFactory");
InitialDirContext idc = new InitialDirContext(p);
Attributes attrs = idc.getAttributes("google.com", new String[] { "A" });
NamingEnumeration<?> ae = attrs.getAll();
while (ae.hasMore()) {
Attribute attr = (Attribute)ae.next();
for (int i = 0; i < attr.size(); i++) {
Object a = attr.get(i);
if (a instanceof String) {
System.out.println(attr.getID() + " " + a);
}
else {
System.out.println(attr.getID() + " NOT ASCII");
}
}
}
ae.close();
I need the TTL alongside the A record. Currently I only get the A address. On checking the response in wireshark each record has its own TTL.
Output
A 142.250.72.238
I'm creating a list of IP address' to ping in which a user can add to the list which is then saved to a properties file in the form of site.name1 = ... site.name2 = ...
Currently I have a for loop with a fixed amount, is there a way to get the number of entries in a properties file so I can set this in the for loop rather than wait for a exception?
PropertiesConfiguration config = configs.properties(new File("IPs.properties"));
//initially check for how many values there are - set to max increments for loop
for (int i = 0; i < 3; i++) { //todo fix
siteName = config.getString("site.name" + i);
siteAddress = config.getString("site.address" + i);
SiteList.add(i, siteName);
IPList.add(i, siteAddress);
}
I've looked through the documentation and other questions but they seem to be unrelated.
It looks to me based on the documentation you should be able to use PropertiesConfiguration#getLayout#getKeys to get a Set of all keys as a String.
I had to modify the code a bit to use apache-commons-configuration-1.10
PropertiesConfiguration config = new PropertiesConfiguration("ips.properties");
PropertiesConfigurationLayout layout = config.getLayout();
String siteName = null;
String siteAddress = null;
for (String key : layout.getKeys()) {
String value = config.getString(key);
if (value == null) {
throw new IllegalStateException(String.format("No value found for key: %s", key));
}
if (key.equals("site.name")) {
siteName = value;
} else if (key.equals("site.address")) {
siteAddress = value;
} else {
throw new IllegalStateException(String.format("Unsupported key: %s", key));
}
}
System.out.println(String.format("name=%s, address=%s", siteName, siteAddress));
I am attempting to enumerate all the groups in our Active Directory from Java. There are quite a lot, so I get a SizeLimitExceededException after 1000 results. I am attempting to use PagedResultsControl, with my code very closely modelled on all the examples out on the web, and it sort-of-works, in that it no longer throws the SizeLimitExceededException, and returns a number of results matching the specified page size (provided that isn't greater than 1000).
However, the next step is to get the cookie from the response and use it to get the next page, and my issue is that there is no PagedResultsResponseControl in the context after the call to search(). In fact getResponseControls() returns null.
I have searched extensively and can't seem to find anyone else reporting this issue, and I'm pretty much stuck here. So what am I doing wrong? Why don't I get a PagedResultsResponseControl?
Our domain is running on Windows Server 2016 and I have reduced my code down to the following test case:
public class PagingTest {
public static void main(String[] args) throws Exception {
Hashtable<String, String> env = new Hashtable<>();
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.SECURITY_PRINCIPAL, "username");
env.put(Context.SECURITY_CREDENTIALS, "password");
env.put(Context.PROVIDER_URL, "ldap://campus.uni.ac.uk/DC=campus,DC=uni,DC=ac,DC=uk");
LdapContext adContext = new InitialLdapContext(env, null);
// Set up search controls.
SearchControls searchControl = new SearchControls();
searchControl.setSearchScope(SearchControls.SUBTREE_SCOPE);
String[] attributesToFetch = {"cn"};
searchControl.setReturningAttributes(attributesToFetch);
// Set up a paged search.
final int pageSize = 500;
byte[] cookie = null;
adContext.setRequestControls(new Control[]{
new PagedResultsControl(pageSize, Control.CRITICAL)
});
// Do the search.
int count = 0;
boolean finished = false;
while (!finished) {
NamingEnumeration<SearchResult> records
= adContext.search("OU=Groups", "objectClass=group", searchControl);
// Examine the page's results control response and act accordingly.
Control[] controls = adContext.getResponseControls();
if (controls != null) {
for (int i = 0; i < controls.length; ++i) {
if (controls[i] instanceof PagedResultsResponseControl) {
PagedResultsResponseControl prrc =
(PagedResultsResponseControl) controls[i];
cookie = prrc.getCookie();
if (cookie == null) {
finished = true;
}
}
}
} else {
cookie = null;
finished = true;
}
// Process the page of results.
while (records != null && records.hasMore()) {
SearchResult sr = records.next();
Attributes attribs = sr.getAttributes();
BasicAttribute ba = (BasicAttribute) attribs.get("cn");
String cn = (String) ba.get();
System.out.println(cn);
++count;
}
// Re-activate paged results with the new cookie.
adContext.setRequestControls(new Control[]{
new PagedResultsControl(pageSize, cookie, Control.CRITICAL)
});
}
System.out.println("Found " + count + " groups");
}
}
Maybe your ldap server does not support paging query, you can use ldapsearch command like this:
ldapsearch -H ldap://xxxx:389 -x -D "uid=zhangsan,ou=employee,dc=test,dc=com" -W -b "" -s base -a always "(objectClass=*)" "supportedControl"
If the return values contain 1.2.840.113556.1.4.319, it shows your ldap server supports paging query.
I have written an application that retrieves Active Directory groups and flattens them, i.e. includes recursively members of subgroup to the top parent group.
It works fine for small groups, but with larger groups I am facing a problem.
If number of members does not exceed 1500, they are listed in the member attribute. If there are more - then this attribute is empty and attribute with name member;range:0-1499 appears, containing first 1500 members.
My problem that I don't know how to get the rest of member set over 1500.
We have groups with 8-12 thousand members. Do I need to run another query?
On the Microsoft site I have seen C# code snippet on the similar matter, but couldn't make much sense of it, as they were showing how to specify a range, but not how to plug it into query. If someone knows how to do it in Java, I'd appreciate a tip.
This will obviously give you the next ones:
String[] returnedAtts = { "member;range=1500-2999" };
You need to fetch the users chunk by chunk (1500 chunks) Just make a counter and update you search and retrieve the next ones until you have all of them.
With your help I have a full working code
// Initialize
LdapContext ldapContext = null;
NamingEnumeration<SearchResult> results = null;
NamingEnumeration<?> members = null;
try {
// Initialize properties
Properties properties = new Properties();
properties.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
properties.put(Context.PROVIDER_URL, "ldap://" + ldapUrl);
properties.put(Context.SECURITY_PRINCIPAL, adminLoginADOnPremise);
properties.put(Context.SECURITY_CREDENTIALS, adminPasswordADOnPremise);
// Initialize ldap context
ldapContext = new InitialLdapContext(properties, null);
int range = 0;
boolean finish = false;
while (finish != true) {
// Set search controls
SearchControls searchCtls = new SearchControls();
searchCtls.setSearchScope(SearchControls.SUBTREE_SCOPE);
searchCtls.setReturningAttributes(generateRangeArray(range));
// Get results
results = ldapContext.search(ldapBaseDn, String.format("(samAccountName=%s)", groupName), searchCtls);
if (results.hasMoreElements() == true) {
SearchResult result = results.next();
try {
members = result.getAttributes().get(generateRangeString(range)).getAll();
while (members.hasMore()) {
String distinguishedName = (String) members.next();
logger.debug(distinguishedName);
}
range++;
} catch (Exception e) {
// Fails means there is no more result
finish = true;
}
}
}
} catch (NamingException e) {
logger.error(e.getMessage());
throw new Exception(e.getMessage());
} finally {
if (ldapContext != null) {
ldapContext.close();
}
if (results != null) {
results.close();
}
}
Two functions missing from the working code example by #Nicolas, I guess they would be something like:
public static String[] generateRangeArray(int i) {
String range = "member;range=" + i * 1500 + "-" + ((i + 1) * 1500 - 1);
String[] returnedAtts = { range };
return returnedAtts;
}
public static String generateRangeString(int i) {
String range = "member;range=" + i * 1500 + "-" + ((i + 1) * 1500 - 1);
return range;
}
The code does not handle the case if the AD group is not so large that the member attribute actually needs to be "chunked", that is if the "member" attribute exists instead.
I want to modify an existing *.rptdesign file and save it under a new name.
The existing file contains a Data Set with a template SQL select statement and several DS parameters.
I'd like to use an actual SQL select statement which uses only part of the DS parameters.
However, the following code results in the exception:
Exception in thread "main" `java.lang.RuntimeException`: *The structure is floating, and its handle is invalid!*
at org.eclipse.birt.report.model.api.StructureHandle.getStringProperty(StructureHandle.java:207)
at org.eclipse.birt.report.model.api.DataSetParameterHandle.getName(DataSetParameterHandle.java:143)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeParamBindingsFor(DataSetHandle.java:851)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeItems(DataSetHandle.java:694)
--
OdaDataSetHandle dsMaster = (OdaDataSetHandle) report.findDataSet("Master");
HashSet<String> bindVarsUsed = new HashSet<String>();
...
// find out which DS parameters are actually used
HashSet<String> bindVarsUsed = new HashSet<String>();
...
ArrayList<OdaDataSetParameterHandle> toRemove = new ArrayList<OdaDataSetParameterHandle>();
for (Iterator iter = dsMaster.parametersIterator(); iter.hasNext(); ) {
OdaDataSetParameterHandle dsPara = (OdaDataSetParameterHandle)iter.next();
String name = dsPara.getName();
if (name.startsWith("param_")) {
String bindVarName = name.substring(6);
if (!bindVarsUsed.contains(bindVarName)) {
toRemove.add(dsPara);
}
}
}
PropertyHandle paramsHandle = dsMaster.getPropertyHandle( OdaDataSetHandle.PARAMETERS_PROP );
paramsHandle.removeItems(toRemove);
What is wrong here?
Has anyone used the DE API to remove parameters from an existing Data Set?
I had similar issue. Resolved it by calling 'removeItem' multiple times and also had to re-evaluate parametersIterator everytime.
protected void updateDataSetParameters(OdaDataSetHandle dataSetHandle) throws SemanticException {
int countMatches = StringUtils.countMatches(dataSetHandle.getQueryText(), "?");
int paramIndex = 0;
do {
paramIndex = 0;
PropertyHandle odaDataSetParameterProp = dataSetHandle.getPropertyHandle(OdaDataSetHandle.PARAMETERS_PROP);
Iterator parametersIterator = dataSetHandle.parametersIterator();
while(parametersIterator.hasNext()) {
Object next = parametersIterator.next();
paramIndex++;
if(paramIndex > countMatches) {
odaDataSetParameterProp.removeItem(next);
break;
}
}
if(paramIndex < countMatches) {
paramIndex++;
OdaDataSetParameter dataSetParameter = createDataSetParameter(paramIndex);
odaDataSetParameterProp.addItem(dataSetParameter);
}
} while(countMatches != paramIndex);
}
private OdaDataSetParameter createDataSetParameter(int paramIndex) {
OdaDataSetParameter dataSetParameter = StructureFactory.createOdaDataSetParameter();
dataSetParameter.setName("param_" + paramIndex);
dataSetParameter.setDataType(DesignChoiceConstants.PARAM_TYPE_INTEGER);
dataSetParameter.setNativeDataType(1);
dataSetParameter.setPosition(paramIndex);
dataSetParameter.setIsInput(true);
dataSetParameter.setIsOutput(false);
dataSetParameter.setExpressionProperty("defaultValue", new Expression("<evaluation script>", ExpressionType.JAVASCRIPT));
return dataSetParameter;
}