I would like to get all the keys from Redis cluster using Jedis with the following code snippet:
public void testRedis() {
String key = "*";
ScanParams scanParams = new ScanParams().count(1000).match("{*}");
String cur = SCAN_POINTER_START;
do {
ScanResult<String> scanResult = getRedisCluster().scan(cur, scanParams);
scanResult.getResult().stream().forEach(System.out::println);
cur = scanResult.getStringCursor();
} while (!cur.equals(SCAN_POINTER_START));
}
My problem is that it doesn't return any result with this solution. Even if I specify the matching pattern for an existing key it's still not working.
I tried to get specific keys with get command it returns value without any error, so the connection seems good.
Any suggestion?
(One of my clue is that match parameter waiting for "curly-brackets" so I had to add there, but I haven't seen using this like that on internet anywhere.)
You can get all keys of a cluster by getting keys from each node and unifying them.
getClusterNodes() method will come in handy which returns a map of all cluster nodes.
Here is an implementation using SCAN, similar to your attempt:
public void testRedis() {
ScanParams scanParams = new ScanParams().count(1000);
Set<String> allKeys = new HashSet<>();
getRedisCluster().getClusterNodes().values().forEach((pool) -> {
String cur = ScanParams.SCAN_POINTER_START;
do {
try (Jedis jedis = pool.getResource()) {
ScanResult<String> scanResult = jedis.scan(cur, scanParams);
allKeys.addAll(scanResult.getResult());
cur = scanResult.getStringCursor();
}
} while (!cur.equals(ScanParams.SCAN_POINTER_START));
});
allKeys.stream().forEach(System.out::println);
}
Update: You can check conditions to stop iteration right after getting at least 1000 keys.
public void testRedis() {
ScanParams scanParams = new ScanParams().count(1000);
Set<String> allKeys = new HashSet<>();
for (JedisPool pool : getRedisCluster().getClusterNodes().values()) {
String cur = ScanParams.SCAN_POINTER_START;
do {
try (Jedis jedis = pool.getResource()) {
ScanResult<String> scanResult = jedis.scan(cur, scanParams);
allKeys.addAll(scanResult.getResult());
cur = scanResult.getStringCursor();
}
if (allKeys.size() >= 1000) break;
} while (!cur.equals(ScanParams.SCAN_POINTER_START));
if (allKeys.size() >= 1000) break;
}
allKeys.stream().forEach(System.out::println);
}
The pattern from the match method should be "*" instead of "{*}"
Related
I'm creating a list of IP address' to ping in which a user can add to the list which is then saved to a properties file in the form of site.name1 = ... site.name2 = ...
Currently I have a for loop with a fixed amount, is there a way to get the number of entries in a properties file so I can set this in the for loop rather than wait for a exception?
PropertiesConfiguration config = configs.properties(new File("IPs.properties"));
//initially check for how many values there are - set to max increments for loop
for (int i = 0; i < 3; i++) { //todo fix
siteName = config.getString("site.name" + i);
siteAddress = config.getString("site.address" + i);
SiteList.add(i, siteName);
IPList.add(i, siteAddress);
}
I've looked through the documentation and other questions but they seem to be unrelated.
It looks to me based on the documentation you should be able to use PropertiesConfiguration#getLayout#getKeys to get a Set of all keys as a String.
I had to modify the code a bit to use apache-commons-configuration-1.10
PropertiesConfiguration config = new PropertiesConfiguration("ips.properties");
PropertiesConfigurationLayout layout = config.getLayout();
String siteName = null;
String siteAddress = null;
for (String key : layout.getKeys()) {
String value = config.getString(key);
if (value == null) {
throw new IllegalStateException(String.format("No value found for key: %s", key));
}
if (key.equals("site.name")) {
siteName = value;
} else if (key.equals("site.address")) {
siteAddress = value;
} else {
throw new IllegalStateException(String.format("Unsupported key: %s", key));
}
}
System.out.println(String.format("name=%s, address=%s", siteName, siteAddress));
Im trying to delete all items in my table in dynamodb but it does not work.
try {
ScanRequest scanRequest = new ScanRequest().withTableName(table);
ScanResult scanResult = null;
do {
if (Check.nonNull(scanResult)) {
scanRequest.setExclusiveStartKey(scanResult.getLastEvaluatedKey());
}
scanResult = client.scan(scanRequest);
scanResult.getItems().forEach((Item) -> {
String n1 = Item.get("n1").toString();
String n2 = tem.get("n2").toString();
DeleteItemSpec spec = new DeleteItemSpec().withPrimaryKey("n1", n1, "n2", n2);
dynamodb.getTable(table).deleteItem(spec);
});
} while (Check.nonNull(scanResult.getLastEvaluatedKey()));
} catch (Exception e) {
throw new BadRequestException(e);
}
n1 is my Primary partition key
n2 is my Primary sort key
The best approach to delete all the items from DynamoDB is to drop the table and recreate it.
Otherwise, there are lot of read capacity and write capacity units being used which will cost you.
Dropping and recreating the table is the best approach.
PREAMBLE: While a scan operation is expensive, I was needing this answer for initialising a table for a test scenario (low volume). The table was being created by another process and I needed the test scenario on that table, I could therefore not delete and recreate the table.
ANSWER:
given:
DynamoDbClient db
static String TABLE_NAME
static String HASH_KEY
static String SORT_KEY
ScanIterable scanIterable = db.scanPaginator(ScanRequest.builder()
.tableName(TABLE_NAME)
.build());
for(ScanResponse scanResponse:scanIterable){
for( Map<String, AttributeValue> item: scanResponse.items()){
Map<String,AttributeValue> deleteKey = new HashMap<>();
deleteKey.put(HASH_KEY,item.get(HASH_KEY));
deleteKey.put(SORT_KEY,item.get(SORT_KEY));
db.deleteItem(DeleteItemRequest.builder()
.tableName(TRANSACTION_TABLE_NAME)
.key(deleteKey).build());
}
}
To delete all the items from the table first you need to perform scan operation over the table which will results you an scanoutcome. Using the iterator loop over the sacnoutcome with the primary key and it's primary key value.This will be one of the approach to delete all the items from the table. Hope that this code will work you. Thanks
Table table = dynamoDB.getTable(your_table);
ItemCollection<ScanOutcome> deleteoutcome = table.scan();
Iterator<Item> iterator = deleteoutcome.iterator();
while (iterator.hasNext()) {
your_table.deleteItem("PrimaryKey", iterator.next().get("primary key value"));
}
//May be we can make it look generic by reading key schema first as below
String strPartitionKey = null;
String strSortKey = null;
TableDescription description = table.describe();
List<KeySchemaElement> schema = description.getKeySchema();
for (KeySchemaElement element : schema) {
if (element.getKeyType().equalsIgnoreCase("HASH"))
strPartitionKey = element.getAttributeName();
if (element.getKeyType().equalsIgnoreCase("RANGE"))
strSortKey = element.getAttributeName();
}
ItemCollection<ScanOutcome> deleteoutcome = table.scan();
Iterator<Item> iterator = deleteoutcome.iterator();
while (iterator.hasNext()) {
Item next = iterator.next();
if (strSortKey == null && strPartitionKey != null)
table.deleteItem(strPartitionKey, next.get(strPartitionKey));
else if (strPartitionKey != null && strSortKey != null)
table.deleteItem(strPartitionKey, next.get(strPartitionKey), strSortKey, next.get(strSortKey));
}
I have TreeMap using the Joda DateTime object and is does not seem to be sorting here is the definition:
TreeMap<DateTime, HolderAnswer> dateTimeTreeMap = new TreeMap<DateTime, HolderAnswer>();
I added in the values as follows (I'm just using a generic sql statement here);
//then get previously selected answers to move to the top of the list
String sql = "Select ActionDT, RecID, TextID, Text, Value from Foo";
Cursor c = DataBaseConnector.query(sql);
if (c != null) {
if (c.moveToFirst()) {
do {
HolderAnswer answer = null;
boolean valueAlreadyIn = false;
DateTime dt = formatter.parseDateTime(c.getString(c.getColumnIndex("ActionDT")));
//we will be adding in the options in the next section, setting to null for now.
answer = new HolderAnswer(c.getInt(c.getColumnIndex("RecID")),c.getInt(c.getColumnIndex("TextID")),null,count,c.getString(c.getColumnIndex("Text")));
//////////////////////////////////////////////////////////////
Iterator<Entry<DateTime, HolderAnswer>> it = dateTimeTreeMap.entrySet().iterator();
while (it.hasNext()) {
Entry<DateTime, HolderAnswer> pairs = it.next();
HolderAnswer tempAnswer = (HolderAnswer) pairs.getValue();
DateTime tempDateTime = (DateTime) pairs.getKey();
//if answers match, transfer over options
if (answer.getTextID() == tempAnswer.getTextID()) {
valueAlreadyIn = true;
}
}
if (!valueAlreadyIn) {
dateTimeTreeMap.put(dt,answer);
}
//////////////////////////////////////////////////////////////////
//count++;
} while(c.moveToNext());
c.close();
c = null;
}
}
When I print out the values, they don't seem to be sorted, they come out in no discernable pattern. Even doing:
dateTimeTreeMap.descendingMap();
Does nothing. Am I missing something?
The descendingMap() method is used to return a reverse order view of the mappings contained in this map so it looks like you're forgetting to assign the sorted map to the original one.
dateTimeTreeMap = dateTimeTreeMap.descendingMap();
I want to modify an existing *.rptdesign file and save it under a new name.
The existing file contains a Data Set with a template SQL select statement and several DS parameters.
I'd like to use an actual SQL select statement which uses only part of the DS parameters.
However, the following code results in the exception:
Exception in thread "main" `java.lang.RuntimeException`: *The structure is floating, and its handle is invalid!*
at org.eclipse.birt.report.model.api.StructureHandle.getStringProperty(StructureHandle.java:207)
at org.eclipse.birt.report.model.api.DataSetParameterHandle.getName(DataSetParameterHandle.java:143)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeParamBindingsFor(DataSetHandle.java:851)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeItems(DataSetHandle.java:694)
--
OdaDataSetHandle dsMaster = (OdaDataSetHandle) report.findDataSet("Master");
HashSet<String> bindVarsUsed = new HashSet<String>();
...
// find out which DS parameters are actually used
HashSet<String> bindVarsUsed = new HashSet<String>();
...
ArrayList<OdaDataSetParameterHandle> toRemove = new ArrayList<OdaDataSetParameterHandle>();
for (Iterator iter = dsMaster.parametersIterator(); iter.hasNext(); ) {
OdaDataSetParameterHandle dsPara = (OdaDataSetParameterHandle)iter.next();
String name = dsPara.getName();
if (name.startsWith("param_")) {
String bindVarName = name.substring(6);
if (!bindVarsUsed.contains(bindVarName)) {
toRemove.add(dsPara);
}
}
}
PropertyHandle paramsHandle = dsMaster.getPropertyHandle( OdaDataSetHandle.PARAMETERS_PROP );
paramsHandle.removeItems(toRemove);
What is wrong here?
Has anyone used the DE API to remove parameters from an existing Data Set?
I had similar issue. Resolved it by calling 'removeItem' multiple times and also had to re-evaluate parametersIterator everytime.
protected void updateDataSetParameters(OdaDataSetHandle dataSetHandle) throws SemanticException {
int countMatches = StringUtils.countMatches(dataSetHandle.getQueryText(), "?");
int paramIndex = 0;
do {
paramIndex = 0;
PropertyHandle odaDataSetParameterProp = dataSetHandle.getPropertyHandle(OdaDataSetHandle.PARAMETERS_PROP);
Iterator parametersIterator = dataSetHandle.parametersIterator();
while(parametersIterator.hasNext()) {
Object next = parametersIterator.next();
paramIndex++;
if(paramIndex > countMatches) {
odaDataSetParameterProp.removeItem(next);
break;
}
}
if(paramIndex < countMatches) {
paramIndex++;
OdaDataSetParameter dataSetParameter = createDataSetParameter(paramIndex);
odaDataSetParameterProp.addItem(dataSetParameter);
}
} while(countMatches != paramIndex);
}
private OdaDataSetParameter createDataSetParameter(int paramIndex) {
OdaDataSetParameter dataSetParameter = StructureFactory.createOdaDataSetParameter();
dataSetParameter.setName("param_" + paramIndex);
dataSetParameter.setDataType(DesignChoiceConstants.PARAM_TYPE_INTEGER);
dataSetParameter.setNativeDataType(1);
dataSetParameter.setPosition(paramIndex);
dataSetParameter.setIsInput(true);
dataSetParameter.setIsOutput(false);
dataSetParameter.setExpressionProperty("defaultValue", new Expression("<evaluation script>", ExpressionType.JAVASCRIPT));
return dataSetParameter;
}
I have verified that the entity I am looking for is in the datastore. I have verified that the list I pass as a method parameter contains this entity. I am trying to find all objects that have their 'userGmail' contained in the list of strings I pass.
Here is my code
#SuppressWarnings("unchecked")
#ApiMethod(name = "findFriendsByEmailList")
public CollectionResponse<ZeppaUser> findFriendsByEmailList(
#Named("emailsList") List<String> emailsList, User user)
throws OAuthRequestException {
if (user == null) {
throw new OAuthRequestException(
"Null User Authorization Exception, findFriendsByEmailList");
}
PersistenceManager mgr = null;
List<ZeppaUser> execute = null;
Query query = null;
try {
mgr = getPersistenceManager();
query = mgr.newQuery(ZeppaUser.class);
query.declareParameters("java.util.List emailListParam");
query.setFilter("emailListParam.contains( userGmail )");
execute = (List<ZeppaUser>) query.execute(emailsList);
query.closeAll();
} finally {
mgr.close();
}
return CollectionResponse.<ZeppaUser> builder().setItems(execute)
.build();
}
This is the stack trace I receive from it:
Something worth noting: I do not receive this error on lists I pass in that to not contain an element found in the datastore. Just when it does exist which leads me to believe that the Query has located the element but has not been closed or executed into a return parameter correctly. If it is preferable to return List that is more than ok. I have tried multiple variations of this with no success thus far. It is getting quite frustrating.
Ok so I found a way around it.
Lists cannot be passed into ApiEndpoints. That or I didn't figure out the correct way to do it and would LOVE an update on the proper way to do this.
Instead, in my client, I construct a String of emails seperated by a comma and send a string into the parameter as an 'encoded' string list then 'decode' it upon execution. Works well but seems hacky.
here are the methods I used. This is convenient though because it works with iOS as well.
public static String encodeListString(ArrayList<String> stringList){
StringBuilder stringbuilder = new StringBuilder();
stringbuilder.append(stringList.get(0));
if(stringList.size() > 1){
for( int i = 0; i < stringList.size(); i++){
stringbuilder.append(",");
stringbuilder.append(stringList.get(i));
}
}
return stringbuilder.toString();
}
public static List<String> decodeListString(String encodedString){
char[] characters = encodedString.toCharArray();
StringBuilder stringbuilder = new StringBuilder();
int position = 0;
ArrayList<String> stringList = new ArrayList<String>();
while(true){
try {
char character = characters[position];
if(character == ','){
String resultString = stringbuilder.toString();
stringList.add(resultString);
stringbuilder = new StringBuilder(); // clear it
} else {
stringbuilder.append(character);
}
position++;
} catch (ArrayIndexOutOfBoundsException aiex){
// List ended
String resultString = stringbuilder.toString();
if(!resultString.isEmpty())
stringList.add(resultString);
break;
}
}
return stringList;
}