Here is a generic question. I'm not in search of the best answer, I'd just like you to express your favourite practices.
I want to implement a network protocol in Java (but this is a rather general question, I faced the same issues in C++), this is not the first time, as I have done this before. But I think I am missing a good way to implement it. In fact usually it's all about exchanging text messages and some byte buffers between hosts, storing the status and wait until the next message comes. The problem is that I usually end up with a bunch of switch and more or less complex if statements that react to different statuses / messages. The whole thing usually gets complicated and hard to mantain. Not to mention that sometimes what comes out has some "blind spot", I mean statuses of the protocol that have not been covered and that behave in a unpredictable way. I tried to write down some state machine classes, that take care of checking start and end statuses for each action in more or less smart ways. This makes programming the protocol very complicated as I have to write lines and lines of code to cover every possible situation.
What I'd like is something like a good pattern, or a best practice that is used in programming complex protocols, easy to mantain and to extend and very readable.
What are your suggestions?
Read up on the State design pattern to learn how to avoid lots of switch statements.
"sometimes what comes out has some "blind spot", I mean statuses of the protocol that have not been covered..."
State can help avoid gaps. It can't guarantee a good design, you still have to do that.
"...as I have to write lines and lines of code to cover every possible situation."
This should not be considered a burden or a problem: You must write lines of code to cover every possible situation.
State can help because you get to leverage inheritance. It can't guarantee a good design, you still have to do that.
Designing a protocol is usually all about the application space you are working within. For instance, http is all about handling web pages, graphics, and posts, while FTP is all about transferring files.
So in short, to start, you should decide what application space you are in, then define the actions that need to be taken. Then finally, before you start designing your actual protocol, you should seriously, seriously hunt for another protocol stack that does what you want to do and avoid implementing a protocol stack altoether. Only after you have determined that something else pre-built absolutely won't work for you should you start building your own protocol stack.
In C++ you can use Boost::Spirit library to parse your protocol message easily. The only "difficulty" is to define the grammar of your message protocol. Take a look at Gnutella source code to see how they solve this problem. Here http://www9.limewire.com/developer/gnutella_protocol_0.4.pdf is the Gnutella protocol specifications
Finite State Machine is what you want
FSM
So you define a whole bunch of states that you can be in as a receiver or sender (idle, connecting_phase1, connecting_phase2, packet expected,...)
Then define all the possible events (packet1 arrives, net closes, ...)
finally you have a table that says 'when in state x and event n happens do func y and transition to state q' - for every state and event (many will be null or dups)
Edit - how to make a FSM (rough sketch)
struct FSMNode
{
int m_nextState;
void (m_func*);
}
FSMNode states[NUMSTATES][NUMEVENTS]=
{ // state 0
{3, bang}, // event 0
{2,wiz},
{1, fertang}
}
{
{1, noop}, // event 0
{1, noop},
{3, ole}
}
.......
FSMNode node = states[mystate][event];
node.m_func(context);
mystate = node.m_nextState;
I am sure this is full of invalid syntax - but I hope you get the drift
Why not use XML as your protocol? You can encapsulate and categorize all your pieces of data inside XML nodes
Can't give you an example myself, but how about looking at how other (competent) people are doing it?
Like this one?
http://anonsvn.jboss.org/repos/netty/trunk/src/main/java/org/jboss/netty/handler/codec/http/
P.S. for that matter, I actually recommend using netty as your network framework and build your protocol on top of it. It should be very easy, and you'll probably get rid of bunch of headaches...
If you are using Java, consider looking at Apache MINA, it's documentation and samples should inspire you in the right way.
Right-click the network connection icon in the System Tray.
Click Troubleshoot problems.
The troubleshooter may find and fix the problem, in this case, you can get quickly started with your business.
If the troubleshooter can't fix the Winsocks problem, then you may get an error looking like:
"One or more network protocols are missing on this computer"
Related
I read the "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in MassiveScale, Unbounded, Out of Order Data Processing" paper. Alas, the SDK does not yet expose the accumulating & retracting triggering mode (section 2.3).
I was wondering if there was a workaround for getting similar semantics?
I have been reading the source and have figured out that StateTag or StateNamespace may be the way i can store the "last emitted value of the window" and hence can be used to calculate the retraction message down the pipeline. Is this the correct path or are there other classes/ways I can/should look at.
The upcoming state API is indeed your best bet for emulating retractions. Those classes you mentioned are part of the state API, but everything in the com.google.cloud.dataflow.sdk.util is for internal use only; we technically make no guarantees that the APIs won't change drastically, or even remain unreleased. That said, releasing that API is on our roadmap, and I'm hopeful we'll get it released relatively soon.
One thing to keep in mind: all the code downstream of your custom retractions will need to be able to differentiate them from normal records. This is something we'll do automatically for you once bonafide retraction support is ready, but in the mean time, you'll just need to make sure all the code you write that might receive a retraction knows how to recognize and handle it as such.
This is one of the questions that involves crossing what I call the "Hello World Gulf" I'm on the "Hello world" I can use SQLite and Content Providers (and resolvers) but I now need to cross to the other side, I cannot make the assumption that onUpgrade will be quick.
Now my go-to book (Wrox, Professional Android 4 development - I didn't chose it because of professional, I chose it because Wrox are like the O'Reilly of guides - O'Reilly suck at guides, they are reference book) only touches briefly on using Loaders, so I've done some searching, some more reading and so forth.
I've basically concluded a Loader is little more than a wrapper, it just does things on a different thread, and gives you a callback (on that worker thread) to process things in, it gives you 3 steps, initiating the query, using the results of the query, and resetting the query.
This seems like quite a thin wrapper, so question 1:
Why would I want to use Loaders?
I sense I may be missing something you see, most "utilities" like this with Android are really useful if you go with the grain so to speak, and as I said Loaders seem like a pretty thin wrapper, and they force me to have callback names which could become tedious of there are multiple queries going on
http://developer.android.com/reference/android/content/Loader.html
Reading that points out that "they ought to monitor the data and act upon changes" - this sounds great but it isn't obvious how that is actually done (I am thinking about database tables though)
Presentation
How should this alter the look of my application? Should I put a loading spinning thing (I'm not sure on the name, never needed them before) after a certain amount of time post activity creation? So the fragment is blank, but if X time elapses without the loader reporting back, I show a spiny thing?
Other operations
Loaders are clearly useless for updates and such, their name alone tells one this much, so any nasty updates and such would have to be wrapped by my own system for shunting work to a worker thread. This further leads me to wonder why would I want loaders?
What I think my answer is
Some sort of wrapper (at some level, content provider or otherwise) to do stuff on a worker thread will mean that the upgrade takes place on that thread, this solves the problem because ... well that's not on the main thread.
If I do write my own I can then (if I want to) ensure queries happen in a certain order, use my own data-structures (rather than Bundles) it seems that I have better control.
What I am really looking for
Discussion, I find when one knows why things are the way they are that one makes less mistakes and just generally has more confidence, I am sure there's a reason Loaders exist, and there will be some pattern that all of Android lends itself towards, I want to know why this is.
Example:
Adapters (for ListViews) it's not immediately obvious how one keeps track of rows (insert) why one must specify a default style (and why ArrayAdapter uses toString) when most of the time (in my experience, dare I say) it is subclasses, reading the source code gives one an understanding of what the Adapter must actually do, then I challenge myself "Can I think of a (better) system that meets these requirements", usually (and hopefully) my answer to that converges on how it's actually done.
Thus the "Hello World Gulf" is crossed.
I look forward to reading answers and any linked text-walls on the matter.
you shouldnt use Loaders directly, but rather LoaderManager
I am trying to design a simple FIX message encoder and decoder to encode (convert to FIX) and decode (convert from FIX) my business domain Order objects. I have designed something, but I am not able to achieve the beautiful design I want. Wanted to see if others who have experience building this kind of things have any better design ideas.
This is what I roughly have: a business Object Order, QuickFIX object Message.
I need to generate NewOrder/Cancel/Replace messages and the message could be different for different exchanges.
I can have ReplaceEncoder --> NewOrderEncoder --> AbstractEncoder, CancelEncoder --> AbstractEncoder.
But if I want another dimension to this, like having custom message generation for different exchanges, then it results in too many combinations of hierarchies.
Is my only bet is to mundanely write different code for different exchanges? How others achieve this? Thanks.
I think you will probably come across a similar problem that we have. That is that each FIX implementation is different. Some use 4.2 others 4.4, some use some tags others ignore them, some use many of their own tags others use very few. What we have done is created general FIX sessions with subclasses for FIX 4.2 and 4.4 and then subclasses for each specific sessions (ie individual brokers). That gives us a reasonable amount of reuse of code for sending and receiving FIX messages. With just the specifics changed for things like handling account names and passwords etc.
For message generation we have a factory method that returns and adapter. All the adapters have the same API which will convert our Business order object in to a FIX Message object. Of course each adapter is specific to the API of the broker. I guess we could probably reuse some code between the adapters but currently we don't.
Is my only bet is to mundanely write different code for different exchanges?
Certainly not. In a FIX message there are compulsory and non compulsory fields. You cannot negotiate on the required fields because then you could not guarantee the authenticity and completeness of the messages. Now I am not saying this is impossible, many counter parties have their own specific user level agreements with exchanges for their own specific messages.
With Quickfix, the XML data dictionary from where the engines confirms the completeness of the messages, is in your hand. Tweak it for your own requirements. You would certainly have multiple sessions. I am not sure if this is possible, haven't tried it myself, does different sessions allow different data dictionaries ? If yes, then use them for different counter parties. If that isn't possible, one way which crosses my mind is add extra code for processing your specific fields, not the whole message, in messages expected from certain counter parties.
One place where I had worked, we were using something on these lines. Receive whatever version you may, but once the message is received convert it into a specific version of FIX message, which only exists inside your system. So your engine basically reads only 1 FIX version of messages. But the added complexity is you have to code a converter. I am not sure how feasible is that for you.
FIX is an extraordinarily slippery protocol when it comes to message definitions.
In practice, every institution that offers a FIX interface has made modifications to the default message set. That means, for instance, a FIX4.4 NewOrderSingle message from counterparty A may have different fields than one from counterparty B.
In fact, counterparty A may have made up some fields whole-cloth and added them in. For any new counterparty, there's a chance you'll encounter fields that you've never seen before.
I've written a few adapters for a few different exchanges, and unfortunately, you're really forced to handle them individually. You may be able to capitalize on some commonalities, but you can't make any assumptions on that until you've reviewed their FIX interface's specs.
So, short answer to your question:
Is my only bet is to mundanely write different code for different exchanges?
Yep, pretty much.
What we ended up doing was writing a base fix layer that applies only the required fix tags. In the fix spec certain tags are flagged as required for each message type.
Once this message had been created we apply a filter to the message that is specific to a broker and instrument type.
ie if you trade options and equities with Goldman and JPMorgan you'd write the following filters:
Goldman-equity
Goldman-option
JPMorgan-Equity
JPMorgan-option
Each would apply vendor and instrument specific fields to the base message.
I'm trying to think in the best way on communication for the game I'm writing. The scenario is simple: tcp sockets and request for authentication, map updates, chat updates, etc. What I was thinking to use was set of classes, like User, Map, Creature, etc and have a Message class, which will have enum with message types and Object to store previously mentioned classes. After I will convert this with GSON to json and on other side I will decode it corresponding to the message type indicated by the element of enum. The problem is that I will pass sometimes too much unnecessary data and that's doesn't let me quiet plus the integration of new types of messages will not be very easy neither for me, nor for someone else who might use it. In the previous version I have used my own XML protocol which also doesn't let me very happy.
So what I'm asking is advice for me the better way for communication or maybe some improvement of my idea.
Thanks in advance,
Serhiy.
XML and JSOn are intended to make application integration simple, but still be human readable.
If you want a protocol tuned to your needs, I suggest you start by determining what information you want to send and how it would look. Document this before you even start implementing it. That way the data sent will suit your needs. (This is more work BTW which is why it is not done more often)
I have an existing Protocol I'd like to write a java Client for. The Protocol consists of messages that have a header containing message type and message length, and then the announced number of bytes which is the payload.
I'm having some trouble modeling it, since creating a Class for each message type seems a bit excessive to me (that would turn out to be 20+ classes just to represent the messages that go over the wire) I was thinking about alternative models. But I can't come up with one that works.
I don't want anything fancy to work on the messages aside from notifying via publish subscribe when a message comes in and in some instances reply back.
Any pointers as to where to look?
A class for each message type is the natural OO way to model this. The fact that there are 20 classes should not put you off. (Depending on the relationship between the messages, you can probably implement common featues in superclasses.)
My advice is to not worry too much about efficiency to start with. Just focus on getting clean APIs that provide the required functionality. Once you've got things working, profile the code and see if the protocol classes are a significant bottleneck. If they are ... then think about how to make the code more efficient.