I use the memcache session handling in AppEngine. Sometimes when doing releases I change objects in a way that renders memcache contents obsolete. When I do and Im testing I want to be able to clear my session.
Ive added a servlet to clear memcache that uses:
try {
CacheFactory cacheFactory = CacheManager.getInstance()
.getCacheFactory();
cacheFactory.createCache(Collections.emptyMap()).clear();
outputMessage(response, "CLEARED cache");
} catch (CacheException e1) {
LOG.log(Level.SEVERE, "cache issue", e1);
outputMessage(response, "cache issue!!!!!!!!");
}
and I dump out the session contents using:
Enumeration<String> e = request.getSession().getAttributeNames();
outputMessage(response, "DUMPING SESSION..");
while (e.hasMoreElements()) {
String name = e.nextElement();
outputMessage(response, "Name:" + name + " value: "
+ request.getSession().getAttribute(name).toString());
}
Doing a dump of session before and after a clear doesnt look any different.
Am I using this right?
Cheers
To get around this problem, I usually append a version id to the key of the object stored in memcache. For example, instead of:
memcache.add('key', 'value')
I do:
version = '1'
memcache.add(VERSION + '#key', 'value')
Later, if I want to invalidate all the data in memcache, I just change the version number (and the entries already stored in memcache will be automatically deleted when they expire).
Related
Apparently, in the move from Spring Boot 1 to Spring Boot 2 (Spring 5), the encoding behavior of URL parameters for RestTemplates changed. It seems unusually difficult to get a general query parameter on rest templates passed so that characters that have special meanings such as "+" get properly escaped. It seems that, since "+" is a valid character, it doesn't get escaped, even though its meaning gets altered (see here). This seems bizarre, counter-intuitive, and against every other convention on every other platform. More importantly, I can't figure out how to easily get around it. If I encode the string first, it gets double-encoded, because the "%"s get re-encoded. Anyway, this seems like it should be something very simple that the framework does, but I'm not figuring it out.
Here is my code that worked in Spring Boot 1:
String url = "https://base/url/here";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), entry.getValue());
}
HttpEntity<TheResponse> resp = myRestTemplate.exchange(builder.toUriString(), ...);
However, now it won't encode the "+" character, so the other end is interpreting it as a space. What is the correct way to build this URL in Java Spring Boot 2?
Note - I also tried this, but it actually DOUBLE-encodes everything:
try {
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), URLEncoder.encode(entry.getValue(),"UTF-8" ));
}
} catch(Exception e) {
System.out.println("Encoding error");
}
In the first one, if I put in "q" => "abc+1#efx.com", then, exactly in the URL, I get "abc+1#efx.com" (i.e., not encoded at all). However, in the second one, if I put in "abc+1#efx.com", then I get "abc%252B1%2540efx.com", which is DOUBLE-encoded.
I could hand-write an encoding method, but this seems (a) like overkill, and (b) doing encoding yourself is where security problems and weird bugs tend to creep in. But it seems insane to me that you can't just add a query parameter in Spring Boot 2. That seems like a basic task. What am I missing?
Found what I believe to be a decent solution. It turns out that a large part of the problem is actually the "exchange" function, which takes a string for a URL, but then re-encodes that URL for reasons I cannot fathom. However, the exchange function can be sent a java.net.URI instead. In this case, it does not try to interpolate anything, as it is already a URI. I then use java.net.URLEncoder.encode() to encode the pieces. I still have no idea why this isn't standard in Spring, but this should work.
private String mapToQueryString(Map<String, String> query) {
List<String> entries = new LinkedList<String>();
for (Map.Entry<String, String> entry : query.entrySet()) {
try {
entries.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" + URLEncoder.encode(entry.getValue(), "UTF-8"));
} catch(Exception e) {
log.error("Unable to encode string for URL: " + entry.getKey() + " / " + entry.getValue(), e);
}
}
return String.join("&", entries);
}
/* Later in the code */
String endpoint = "https://baseurl.example.com/blah";
String finalUrl = query.isEmpty() ? endpoint : endpoint + "?" + mapToQueryString(query);
URI uri;
try {
uri = new URI(finalUrl);
} catch(URISyntaxException e) {
log.error("Bad URL // " + finalUrl, e);
return null;
}
}
/* ... */
HttpEntity<TheResponse> resp = myRestTemplate.exchange(uri, ...)
I am using UnboundID-LDAPSDK (2.3.8) to change the user's photo in our Microsoft Active Directory.
LDAPConnection ldap = null;
try {
ldap = new LDAPConnection("domain-srv", 389, "CN=admin,OU=Users,OU=ADM,DC=domain,DC=local", "password");
SearchResult sr = ldap.search("DC=domain,DC=local", SearchScope.SUB, "(sAMAccountName=" + getUser().getUsername() + ")");
if (sr.getEntryCount() == 1) {
SearchResultEntry entry = sr.getSearchEntries().get(0);
entry.setAttribute("thumbnailPhoto", getUser().getPhotoAsByteArray());
ldap.close();
return true;
} else
return false;
} catch (LDAPException e) {
e.printStackTrace();
}
But I get a java.lang.UnsupportedOperationException.
The documentation for setAttribute states:
Throws an UnsupportedOperationException to indicate that this is a
read-only entry.
I also tried to change the postalCode but I get the same exception.
Changing those attributes should be possible, because I can change them with jXplorer.
Do I have to enable a write-mode somehow?
Thank you
The SearchResultEntry object extends ReadOnlyEntry and is therefore immutable. But even if it weren't, merely calling entry.setAttribute would have no effect on the data in the server. You have to use a modify operation for that.
To do that, you'd need something like:
ModifyRequest modifyRequest = new ModifyRequest(entry.getDN(),
new Modification(ModificationType.REPLACE,
"thumbnailPhoto", getUser().getPhotoAsByteArray());
ldap.modify(modifyRequest);
Also, you should put the call to ldap.close() in a finally block because as the code is written now, you're only closing the connection if the search is successful and returns exactly one entry, but not if the search fails, doesn't match any entries, or the attempt to perform the modify fails.
I am retrieving account information from an LDAP/AD via JNDI from an agent in Lotus Notes. That works just fine. My problem are the atrributes/values with equal names e.g. there are ObjectClass and ObjectType which are availible multiple times in the entry. I want to store the values as a multitext entry in LotusNotes with the following code
for (NamingEnumeration ae = atts.getAll(); ae.hasMore();) {
Attribute attr = (Attribute) ae.next();
Item mItem = newDoc.getFirstItem(fPre + "_" + attr.getID());
if(mItem == null){
newDoc.replaceItemValue(fPre + "_" + attr.getID(), attr.get());
}else{
mItem.appendToTextList("" + attr.get());
}
}
But I only get always one value. So do I accessing the entry in the LDAP/AD the wrong way or is the storing in Notes wrong (is not possible to access the Item without saving, like it is possible with LotusScript)?
Thanks
Armin
I have verified this multiple times using appstats. When the below code is NOT wrapped in a transaction, JDO performs two datastore reads and one write, 3 RPC's, at a cost of 240. Not just the first time, every time, even though it is accessing the same record every time hence should be pulling it from cache. However, when I wrap the code in a transaction as above, the code makes 4 RPC's: begin transaction, get, put, and commit -- of these, only the Get is billed as a datastore read, so the overall cost is 70.
If it's pulling it from cache, why would it only bill for a read? It would seem that it would bill for a write, not a read. Could app engine be billing me the same amount for non-transactional cache reads as it does for datastore reads? why?
This is the code WITH transaction:
PersistenceManager pm = PMF.getManager();
Transaction tx = pm.currentTransaction();
String responsetext = "";
try {
tx.begin();
Key userkey = obtainUserKeyFromCookie();
User u = pm.getObjectById(User.class, userkey);
Key mapkey = obtainMapKeyFromQueryString();
// this is NOT a java.util.Map, just FYI
Map currentmap = pm.getObjectById(Map.class, mapkey);
Text mapData = currentmap.getMapData(); // mapData is JSON stored in the entity
Text newMapData = parseModifyAndReturn(mapData); // transform the map
currentmap.setMapData(newMapData); // mutate the Map object
tx.commit();
responsetext = "OK";
} catch (JDOCanRetryException jdoe) {
// log jdoe
responsetext = "RETRY";
} catch (Exception e) {
// log e
responsetext = "ERROR";
} finally {
if (tx.isActive()) {
tx.rollback();
}
pm.close();
}
resp.getWriter().println(responsetext);
This is the code WITHOUT the transaction:
PersistenceManager pm = PMF.getManager();
String responsetext = "";
try {
Key userkey = obtainUserKeyFromCookie();
User u = pm.getObjectById(User.class, userkey);
Key mapkey = obtainMapKeyFromQueryString();
// this is NOT a java.util.Map, just FYI
Map currentmap = pm.getObjectById(Map.class, mapkey);
Text mapData = currentmap.getMapData(); // mapData is JSON stored in the entity
Text newMapData = parseModifyAndReturn(mapData); // transform the map
currentmap.setMapData(newMapData); // mutate the Map object
responsetext = "OK";
} catch (Exception e) {
// log e
responsetext = "ERROR";
} finally {
pm.close();
}
resp.getWriter().println(responsetext);
With the transaction, the PersistenceManager can know that the caches are valid throughout the processing of that code. Without the transaction, it cannot (it doesn't know whether some other action has come in behind its back and changed things) and so must validate the cache's contents against the DB tables. Each time it checks, it needs to create a transaction to do so; that's a feature of the DB interface itself, where any action that's not in a transaction (with a few DB-specific exceptions) will have a transaction automatically added.
In your case, you should have a transaction anyway, because you want to have a consistent view of the database while you do your processing. Without that, the mapData could be modified by another operation while you're in the middle of working on it and those modifications would be silently lost. That Would Be Bad. (Well, probably.) Transactions are the cure.
(You should also look into using AOP for managing the transaction wrapping; that's enormously easier than writing all that transaction management code yourself each time. OTOH, it can add a lot of complexity to deployment until you get things right, so I could understand not following this piece of advice…)
When I am creating a new H2 database via ORMLite the database file get created but after I close my application, all the data that it stored in the database is lost:
JdbcConnectionSource connection =
new JdbcConnectionSource("jdbc:h2:file:" + path.getAbsolutePath() + ".h2.db");
TableUtils.createTable(connection, SomeClass.class);
Dao<SomeClass, Integer> dao = DaoManager.createDao(connection, SomeClass.class);
SomeClass sc = new SomeClass(id, ...);
dao.create(sc);
SomeClass retrieved = dao.queryForId(id);
System.out.println("" + retrieved);
This code will produce good results. It will print the object that I stored.
But when I start the application again this time without creating the table and storing new object I get an exception telling me that the required table is not exists:
JdbcConnectionSource connection =
new JdbcConnectionSource("jdbc:h2:file:" + path.getAbsolutePath() + ".h2.db");
Dao<SomeClass, Integer> dao = DaoManager.createDao(connection, SomeClass.class);
SomeClass retrieved = dao.queryForId(id); // will produce an exception..
System.out.println("" + retrieved);
The following worked fine for me if I ran it once and then a second time with the createTable turned off. The 2nd insert gave me a primary key violation of course but that was expected. It created the file with (as #Thomas mentioned) a ".h2.db.h2.db" prefix.
Some questions:
After you run your application the first time, can you see the path file being created?
Is it on permanent storage and not in some temporary location cleared by the OS?
Any chance some other part of your application is clearing it before the database code begins?
Hope this helps.
#Test
public void testStuff() throws Exception {
File path = new File("/tmp/x");
JdbcConnectionSource connection = new JdbcConnectionSource("jdbc:h2:file:"
+ path.getAbsolutePath() + ".h2.db");
// TableUtils.createTable(connection, SomeClass.class);
Dao<SomeClass, Integer> dao = DaoManager.createDao(connection,
SomeClass.class);
int id = 131233;
SomeClass sc = new SomeClass(id, "fopewjfew");
dao.create(sc);
SomeClass retrieved = dao.queryForId(id);
System.out.println("" + retrieved);
connection.close();
}
I can see Russia from my house:
> ls -l /tmp/
...
-rw-r--r-- 1 graywatson wheel 14336 Aug 31 08:47 x.h2.db.h2.db
Did you close the database? It is closed automatically but it's better to close it manually (so recovery is faster).
In many cases the database URL is the problem. Are you sure the same path is used in both cases? Otherwise you end up with two databases. By the way, ".h2.db" is added automatically, you don't need to add it manually.
To better analyze the problem, you could append ;TRACE_LEVEL_FILE=2 to the database URL, and then check in the *.trace.db file what SQL statements were executed against the database.