Recently I have been investigating on feasibility of using Zookeeper (super excited about it!!! :) ) for a existing platform.
Existing Architecture:
Briefly, the module does a bunch of stuff from
Concerns:
In my next blog, I demonstrate few features which can be used to correct above fallacies like Leader Election, Configuration management, Distributed Locking, Group Membership.
Existing Architecture:
Briefly, the module does a bunch of stuff from
- Registering itself when instance is started.
- Work delegation among other instances through rpc
- Acquiring lock for doing work (note due to distributed nature, is it stored outside of VM)
Concerns:
- Synchronous remote method invocation: When one instance delegates work to another, the request goes through Spring's HttpInvoker which under the hood uses apache commons HttpClient. Since the underlying methods are either a GET/POST, I believe the remote method invocation results in synchronous calls effectively blocked until results are retrieved. The remote calls are expensive ( atleast 4-5 times than a normal method invocation). More details can be found in the paper. Currently it works because there are two production instances invoking each other and it has to deal with transactions in the range 17-20k a day. Looking 6 months back we have doubled our processing volume, continuing that trend, in a years time we might be at 80k transactions a day. Will our current solution be sufficient for scale?
- Fail over / High availability: Both instances are parallel running and both try to do the same work. That is both are configured to run in Master mode. The way to do any work to avoid duplicate is by acquiring a shared lock. Thus High Availability is achieved through replication but it is not the right way to do it. In a ideal situation, they should be configured as Master/Slave with the ability to take over by Slave in case if Master fails.
- Locking: Locks are granulated based on the type of work and are also classified into blocking/non-blocking locks. Concerning part is they are acquired and released everytime when a work is done. More instances, more of this, which will increase the load on database.
- Lastly, this allocation part makes the system highly coupled and confusing.
In my next blog, I demonstrate few features which can be used to correct above fallacies like Leader Election, Configuration management, Distributed Locking, Group Membership.
Hi Amrut, I had attempted to use ZK couple of years back (and gave up) in order to control a bunch of services which would be available to service a queue of jobs. The idea was to use a routing server to assign jobs to one of the processing servers using hashmod partitioning. The idea of using ZK was that if one (or more) of the servers died, the hashmod partitioning would be modified to reflect that there were fewer servers available now. However, I was unable to figure out how to make the ZK and the other servers recognize and interoperate with each other. We ultimately moved to a slightly different queueing strategy where the servers pulled the next available task from the queue rather than a router assigning it to them. Any pointers on how the first approach could be done using ZK? Thanks
ReplyDeleteFor noticing state change:
ReplyDelete1) Register the processing servers as nodes in ZK datatree EPHEMERAL_SEQUENTIAL. For eg: /SERVER/NODE-.
2) Set watcher either.
a) At NODE- level: Set watch on the previous NODE- node to avoid the herd effect. Handle when in process method for Event.EventType.
b) At SERVER level: Set watch on PARENT node and handle when its children (NODE-) change.
This seems to be somewhat on the lines of GROUP MEMBERSHIP: http://amrutbudihal.blogspot.com/2012/07/zookeeper-leader-election-and-group.html
Finally got it after reading through chapter 15 of Mahout in Action. You need to set a watcher in the client to listen to watch events thrown by Zookeeper (thats what you were saying in 2a above) and do application level things in response to the event.
ReplyDelete