I am building a RSS reader using the Play Framework, play authenticate, and mongodb accessed through Morphia. It's running on a Ubuntu server with a VIA Nano processor U2250 (1.6GHz Capable) and 2ghz ram apparently. Despite every tweak and optimization I could think of, I still observe insane delays in the answering time: from 2seconds to 1 minute to answer a simple HTTP GET request to mark an item as read (I'll talk about this example because it is simple). Note that i am not facing this problem on local tests on my personal machine, just on remote deployment.
Profiling gives me the following distribution of spent time to answer a dozen "mark as read" requests :
org.bson.io.PoolOutputBuffer.pipe() 11%
org.bson.io.Bits.readFully() 10%
scala.concurrent.forkjoin.ForkJoinPool.scan() 8%
org.jboss.netty.channel.socket.nio.SelectorUtil.select() 7%
org.bson.io.PoolOutputBuffer.write 5%
com.google.code.morphia.mapping.DefaultCreator.getNoArgsConstructor() 5%
from which I gathered the I/O with the database was the bottleneck. I have made the references explicit using a custom class Ref<> because Key<> was handling package names fairly poorly. Therefore the aforementioned mark as read request only does 3 db queries : retrieve the item by its ref (id), retrieve the inbox by its ref (id) stored in the session cache, remove the item from the inbox and store it back again. My interactions with the database are encapsulated in a static helper class :
public class MorphiaHelper {
static private Mongo mongo;
static private Morphia morphia;
static private Datastore datastore;
public static void mapModels() {
morphia.map(...);
}
public static void startup() {
try {
mongo = new MongoClient();
morphia = new Morphia();
datastore = MorphiaHelper.morphia.createDatastore(MorphiaHelper.mongo, "dbname");
mapModels();
updateModelsIfNeeded();
datastore.ensureIndexes();
datastore.ensureCaps();
} catch (Exception e) {
Logger.error("Database failed to initialize:" + e.getMessage());
}
Logger.debug("** mongo and morphia initialization on datastore: " + MorphiaHelper.datastore.getDB());
}
public static void shutdown() {
mongo.close();
Logger.debug("** mongo and morphia closed **");
}
public static <T> Key<T> save(T entity){
return datastore.save(entity);
}
public static <T, V> T get(Ref<V> ref) throws NotFoundException{
try {
Class<T> classT = (Class<T>)Class.forName(ref.getClassName());
return datastore.get(classT, ref.getId());
} catch (ClassNotFoundException e) {
throw new NotFoundException("Reference to morphia not found:" + ref.toString());
}
}
...
}
and rely on the aforementionned inner static Ref class :
public static interface Referencable{
Ref getRef();
}
public static class Ref<T> implements Serializable, Comparable {
Map<String, String> id;
private static String nameToKey(String name){
return name.replace(".", " ");
}
private static String keyToName(String key){
return key.replace(" ", ".");
}
public static <T> Ref<T> fromIdString(Class<T> className, String idString) {
return new Ref<T>(className, idString);
}
public static Ref<?> fromString(String sourceRefString) throws ClassNotFoundException {
if (! sourceRefString.contains("=")){
throw new ClassNotFoundException();
}
String[] parseString = sourceRefString.split("=", 2);
Class<?> className = Class.forName(keyToName(parseString[0]));
return fromIdString(className, parseString[1]);
}
#Deprecated
private Ref() {}
public Ref(Class<T> _className, String _id) {
id = Collections.singletonMap(nameToKey(_className.getName()), _id);
}
private Entry<String, String> getIdFromMap(){
return id.entrySet().iterator().next();
}
public Object getId(){
return getIdFromMap().getValue();
}
public String toString(){
return getIdFromMap().toString();
}
#Override
public boolean equals(Object other){
return (other.getClass().equals(this.getClass())
&& ((Ref)other).getIdFromMap().getKey().equals(this.getIdFromMap().getKey())
&& ((Ref)other).getIdFromMap().getValue().equals(this.getIdFromMap().getValue()));
}
#Override
public int hashCode() {
return getIdFromMap().hashCode();
}
public String getClassName() {
return keyToName(getIdFromMap().getKey());
}
#Override
public int compareTo(Object arg0) {
return this.toString().compareTo(((Ref) arg0).toString());
}
}
Since I haven't been able to clearly locate the problem, I am also offering you just in case the code of the controller :
public static Result read(final String id) throws NotFoundException{
Item item = MorphiaHelper.get(Ref.fromIdString(Item.class, id));
Application.getLocalInbox().remove(item);
return ok();
}
This is Application, getting us the inbox :
public class Application extends Controller {
public static Inbox getLocalInbox() throws NotFoundException {
return Inbox.getInboxOfUser(getLocalUserRef());
}
public static User getLocalUser() throws NotFoundException {
User user = UserDAO.findUser(PlayAuthenticate.getUser(session()));
session("UserRef", user.getRef().toString());
return user;
}
public static Ref<User> getLocalUserRef() throws NotFoundException {
if (session("UserRef") == null){
session("UserRef", getLocalUser().getRef().toString());
}
try {
return (Ref<User>) Ref.fromString(session("UserRef"));
} catch (Exception e){
throw new NotFoundException("could not retrieve current user ref");
}
}
}
So yeah nothing very interesting here. This is inbox.remove :
public void remove(Item item) {
// TODO can be optimized if we dont check the existence of tags
boolean listContainedElement = false;
for(Ref<Tag> tag : item.getTagsRef()){
List<Ref<Item>> list = inbox().get(tag);
if (list != null){
Ref<Item> key = item.getRef();
boolean remove = list.remove(key);
listContainedElement = listContainedElement || remove;
if (list.size() == 0){
inbox().remove(tag);
}
}
}
List<Ref<Item>> list = inbox.get(Tag.getUntagged());
if (list != null){
listContainedElement = listContainedElement || list.remove(item.getRef());
}
if (listContainedElement) seen(item);
MorphiaHelper.save(this);
}
where
Map<String, List<Ref<Item>>> inbox = new HashMap<String, List<Ref<Item>>>();
Map<Ref<Tag>, List<Ref<Item>>> inbox(){
return new MorphiaMap(inbox);
}
is a way to bypass the fact that morphia cannot handle anything else than Strings as map keys and keep code clean, MorphiaMap being a class storing an attribute "Map innerMap;" and forwarding all the commands with the String to Ref translation.
I'm completely at a loss here so any kind of advice would be much appreciated. I've been tweaking the system for weeks to improve performances but I'm afraid I'm missing the elephant in the room.
Thank you in advance
Best regards
Related
Hi
In the game Minecraft you have to send data between the client and the server in order to sync stuff, one of the things that need to be synced in particular circumstances is TileEntities with a render. The data is stored on the server and sent to the client which is thereafter used for render.
You create a class that contains the data that needs to be synced and attach it to a channel creation API called "SimpleImpl". My Network object and a Message Object:
public class IntercraftPacketHandler
{
private static int index = 1;
private static final ResourceLocation CHANNEL_NAME = new ResourceLocation(Reference.MODID,"network");
private static final String PROTOCOL_VERSION = new ResourceLocation(Reference.MODID,"1").toString();
public static SimpleChannel getNetworkChannel()
{
final SimpleChannel channel = NetworkRegistry.ChannelBuilder.named(CHANNEL_NAME)
.clientAcceptedVersions(version -> true)
.serverAcceptedVersions(version -> true)
.networkProtocolVersion(() -> PROTOCOL_VERSION)
.simpleChannel();
// Sync Capability Identity Hidden data message.
channel.messageBuilder(MessageIdentityHidden.class,index)
.encoder(MessageIdentityHidden::encode)
.decoder(MessageIdentityHidden::decode)
.consumer(MessageIdentityHidden::handle)
.add(); index++;
// Send TreeTapTileEntity data to client.
channel.messageBuilder(MessageTreeTap.class,index)
.encoder(MessageTreeTap::encode)
.decoder(MessageTreeTap::decode)
.consumer(MessageTreeTap::handle)
.add(); index++;
// Send ChunkLoaderTileEntity data to client.
channel.messageBuilder(MessageChunkLoader.class,index)
.encoder(MessageChunkLoader::encode)
.decoder(MessageChunkLoader::decode)
.consumer(MessageChunkLoader::handle)
.add(); index++;
return channel;
}
}
public class MessageChunkLoader
{
private BlockPos pos;
private boolean canLoad;
public MessageChunkLoader(BlockPos pos,boolean canLoad)
{
this.pos = pos;
this.canLoad = canLoad;
}
public void handle(Supplier<NetworkEvent.Context> ctx)
{
ctx.get().enqueueWork(() -> {
try {
ChunkLoaderBaseTileEntity tile = (ChunkLoaderBaseTileEntity) Minecraft.getInstance().world.getTileEntity(pos);
tile.canLoad = canLoad;
} catch (NullPointerException err) {
System.out.println(String.format("Could not find ChunkLoaderTileEntity at %s %s %s!",pos.getX(),pos.getY(),pos.getZ()));
}
});
}
public static void encode(MessageChunkLoader message, PacketBuffer buffer)
{
buffer.writeBlockPos(message.pos);
buffer.writeBoolean(message.canLoad);
}
public static MessageChunkLoader decode(final PacketBuffer buffer)
{
return new MessageChunkLoader(buffer.readBlockPos(),buffer.readBoolean());
}
}
I then initialize it in my main mod class used by objects in my mod project.
#Mod(Reference.MODID)
public class IntercraftCore
{
public static final SimpleChannel NETWORK = IntercraftPacketHandler.getNetworkChannel();
...
The problem and this post's question; right now I create a new message class from the formula I follow in MessageChunkLoader (public static encode & decode method and a handle method). I would like to create a more generic class for creating message classes for TileEntities, but I'm having problems with that. Here's the current class:
public abstract class MessageTileEntity<T extends TileEntity>
{
protected final BlockPos pos;
protected final Class<T> clazz;
public MessageTileEntity(BlockPos pos, Class<T> clazz)
{
this.pos = pos;
this.clazz = clazz;
}
public abstract void handle(Supplier<NetworkEvent.Context> ctx);
protected T getTileEntity()
{
try {
return clazz.cast(Minecraft.getInstance().world.getTileEntity(pos));
} catch (NullPointerException e) {
System.out.println(String.format("Could not find %s at [%d %d %d]!",clazz.getSimpleName(),pos.getX(),pos.getY(),pos.getZ()));
throw e;
} catch (ClassCastException e) {
System.out.println(String.format("TileEntity at [%d %d %d] is not %s!",pos.getX(),pos.getY(),pos.getZ(),clazz.getSimpleName()));
throw e;
}
}
public static void encode(MessageTileEntity message, PacketBuffer buffer)
{
}
public static MessageTileEntity decode(final PacketBuffer buffer)
{
return null;
}
}
The main problem is I lack the proper Java skills to make it like I want it to function. The method handle is easy as it's non-static and needs to be custom to every TileEntity message, but the methods encode and decode which needs to be static gives me problems. I have no idea what I'm trying to achieve is possible, asking won't hurt. Maybe the solution is easier than I think.
My requirement is something like this:
I have written a code which checks for data in database and if data is not found, it consumes third party rest API in order to get data. Some logic is performed and finally obtained data is stored in database.
public void exampleMethod(){
MyObject myObj = getObjFromDatabase(); //getting from db
if(myObj==null){ //not found in db
myObj = getObjFromThirdParty(); //consuming rest api
//some logic here in case myobj is not in db....
}else{
//some other logic here in case myobj is in db...
}
saveObjInDatabase(myObj); //saving it in database
}
I need it to be saved in database just once. Getting response from third party API takes some time and this method gets executed from multiple threads.
Now, the problem is I need it to be saved in database just once, but before one thread could save data in database another thread gets null from database, logic that should get executed only when "data is not in db" gets executed and saves same data more than once. (I am using mongoDB to store data)
How can I solve this problem? Thank you.
What you are asking about is caching. Here is an example that should work decently. It synchronizes on getObj in case the object needs to be loaded. If the object is fresh then getObj returns very quickly and hardly blocks the other threads, but if it needs to load the object then the other threads will wait until the object is loaded.
public class Test {
// used when you want to refresh the cached object
private static boolean refreshNext = false;
// a static reference to your object that all threads can use
private static MyObj cachedObj = null;
private static void message(String msg) {
System.out.println(
System.currentTimeMillis() + " : " + Thread.currentThread().getName() + " : " + msg);
}
private static void sleep(long milli) {
try { Thread.sleep(milli); } catch (Exception e) { }
}
// represents the object stored in the db
private static MyObj myObjInDb = null;
private static void saveObjInDb(MyObj obj) {
// TODO do real saving to db and handle errors
message("storing in db...");
myObjInDb = obj;
}
private static MyObj loadObjFromDb() {
// TODO do real loading from db and handle errors
message("getting from db...");
sleep(1000);
return myObjInDb;
}
private static MyObj loadObjFromVendor() {
// TODO do real fetching from vendor and handle errors
message("getting from vendor...");
sleep(2000);
return new MyObj();
}
private static MyObj loadObj() {
message("loading object...");
MyObj obj = loadObjFromDb();
if (obj == null) {
message("db didn't have it.");
obj = loadObjFromVendor();
saveObjInDb(obj);
}
return obj;
}
/** Returns the object, first loading and caching if needed. */
public static synchronized MyObj getObj() {
// only one thread can get the object at a time, in case it needs to be loaded.
if (cachedObj == null || refreshNext) {
// load and cache the object
cachedObj = loadObj();
refreshNext = false;
}
return cachedObj;
}
public static void exampleMethod() {
MyObj obj = getObj();
message(obj.toString());
// ... do stuff with obj
}
private static class MyObj {
public final String data = "I have data!";
#Override public String toString() { return data; }
}
public static void main(String[] args) {
for (int i = 0; i < 20; i++)
new Thread(Test::exampleMethod).start();
}
}
I have the following code which build a combobox of states. The code works as expected however I can not get auto complete to work correctly. It appears that the autocomplete is returning object id's instead of the expected text. Clicking on the drop down shows the correct abbreviated state names. Where am I going wrong?
private void buildStateCombo() {
List<StatesDTO> states = GetStateList.getStatesList();
StateCombo.setItems(FXCollections.observableList(states));
StateCombo.setEditable(true);
TextFields.bindAutoCompletion(StateCombo.getEditor(), StateCombo.getItems());
/**
* StringConverter
*/
StateCombo.setConverter(new StringConverter<StatesDTO>() {
#Override
public String toString(StatesDTO object) {
if (object == null){
return null;
} else {
return object.getStateShort();
}
}
#Override
public StatesDTO fromString(String string) {
return DriverAddStateCombo.getItems().stream().filter(state ->
state.getStateShort().equals(string)).findFirst().orElse(null);
}
});
}
GetStateList look like this...
public class GetStateList {
public static List<StatesDTO> getStatesList() {
Database db = new Database();
List<StatesDTO> statesList = new ArrayList<>();
try {
String query = "SELECT stateID, stateCode FROM States";
ResultSet statesRS = db.executeQuery(query);
while(statesRS.next()) {
StatesDTO statesdto = new StatesDTO();
statesdto.setStateID(statesRS.getString(1));
statesdto.setStateShort(statesRS.getString(2));
statesList.add(statesdto);
}
} catch (SQLException ex) {
System.out.println(ex.getMessage());
}
db.closeConnection();
return statesList;
}
}
And StatesDTO
public class StatesDTO {
public SimpleStringProperty stateID = new SimpleStringProperty();
public SimpleStringProperty stateShort = new SimpleStringProperty();
public String getStateID() {
return stateID.get();
}
public void setStateID(String stateIDStr) {
stateID.set(stateIDStr);
}
public String getStateShort() {
return stateShort.get();
}
public void setStateShort(String stateShortStr) {
stateShort.set(stateShortStr);
}
}
One way I could make this work to provide the names directly in a collection.
TextFields.bindAutoCompletion(StateCombo.getEditor(),
StateCombo.getItems().stream().map(state ->
state.getStateShort()).collect(Collectors.toList()) );
I tried different signatures of the method, such as
bindAutoCompletion(TextField textField, Callback> suggestionProvider, StringConverter converter)
but I couldn't make it work.
Note: ControlsFX version 8.40.12.
Scenario 1 :
I am generating a report for more department's performance and participation in a institute. When I am display the report in GUI, it can be sort by department performance and participation(No.of student participated).
For this scenario, should i use Prototype Design pattern?
Ex :
public abstract class Report implements Cloneable {
private String id;
protected String type;
public void setId(String id){
id=id;
}
public String getId(){
return id;
}
public String getType(){
return type;
}
abstract void getReportData();
public Object clone() {
Object clone = null;
try {
clone = super.clone();
} catch (CloneNotSupportedException e) {
e.printStackTrace();
}
return clone;
}
}
public class PerformanceReport extends Report {
public PerformanceReport(){
type = "Performance";
}
#Override
public void getReportData() {
/* Get report data from database and sort based on performance*/
}
}
public class ParticipationReport extends Report {
public ParticipationReport(){
type = "Participation";
}
#Override
public void getReportData() {
/* Get report data from database and sort based on participation*/
}
}
public class ReportCache {
private static Hashtable<String, Report> reportMap = new Hashtable<String, Report>();
public static Report getReport(String reportid) {
Report cachedReport = reportMap.get(reportid);
return (Report) cachedReport.clone();
}
public static void loadCache() {
ParticipationReport participationReport = new ParticipationReport();
participationReport.setId("1");
reportMap.put(report.getId(),report);
PerformanceReport performanceReport = new PerformanceReport();
performancenReport.setId("2");
reportMap.put(report.getId(),report);
}
}
public class PrototypePatternReport {
public static void main(String[] args) {
ReportCache.loadCache();
Report clonedReport = (Report) ReportCache.getReport("1");
System.out.println("Report : " + clonedReport.getType());
Report clonedReport2 = (Report) ReportCache.getReport("2");
System.out.println("Report : " + clonedReport2.getType());
}
}
Is my above concept is correct ? and this concept is relevant to Prototype-pattern?
Scenario 2 :
I am storing quiz detail (questions and options, answers) in a object, while student request for quiz, I should encrypt the answer and give. For encrypted answer i should keep another object to give. I this scenario can i use prototype? After response come from student I should compare the student answer with existing object.
Prototype pattern is often useful when object initialization is expensive or when you explicitly need an object that is a copy of another.
Scenario 1:
In your case, getting report data from database and sorting it is much more expensive than instantiating an object, and each report will consist on its own data (you will not benefit from copying from another object) so I would not consider using a prototype.
Scenario 2:
In this scenario, the key is
For encrypted answer i should keep another object to give
In this case, as you need another object and you need to ensure that the second object is an exact copy of the first, you could use a prototype to create the second object, and then change its properties to ensure that the answers are hidden.
I want to design API which could handle XPATH input from user.
Currently i have model the XPATH input in following way,
public interface ICondition {
String getConditionString();
}
public class XPathCondition implements ICondition {
private Class<? extends XPATHFunction> clazz = null;
private Operator operator = null;
private String compValue = null;
private String param = null;
public void setXPathFunction(Class<? extends XPATHFunction> clazz) {
this.clazz = clazz;
}
public void setComparisionType(Operator operator) {
this.operator = operator;
}
public void setComparisionValue(String value) {
this.compValue = value;
}
public void setParam(String param) {
this.param = param;
}
public String getConditionString() {
XPATHFunction function = null;
try {
function = (XPATHFunction) clazz.newInstance();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
return function.call(param) + operator.getOprValue() + compValue;
}
public static void main(String[] args) {
XPathCondition xpathCond = new XPathCondition();
xpathCond.setXPathFunction(CountFunction.class);
xpathCond.setParam("/CPRRegistrationInfo/*");
xpathCond.setComparisionType(Operator.GT);
xpathCond.setComparisionValue("0");
System.out.println(xpathCond.getConditionString());
}
}
public interface XPATHFunction {
public String call(String param);
}
public class CountFunction implements XPATHFunction {
public String call(String param) {
return "count(" + param + ") ";
}
}
There could be other XPATH function which have to implement and interface XPATHFunction and implement it in its way.
API just have create XPATHCondition and set appropriate function and call getConditionString() method to get the final xpath.
Is there any better way, we can model XPATH input?
Please help me to re factor the above design.
I don't think I cannot really comment on your class structure or suggest something different. It is a very tough problem.
What I'd suggest is:
Start with some simple sub-set of the full XPath syntax
Instead of starting from the design of the API and the classes start designing the user interface: what are you going to show the user? What are his/her possible actions? The class design should then derive from the UI design.
If possible use a schema for the underlying XML, so that you can present the user with a limited choice of the possible element and attribute name to use in the expression