Race Condition on start

[behemphi]

There appears to be a race condition where, if the image already exists on the cluster, then the containers start fast enough that between the time the role is selected and the time the machine comes up what is reported in etcd is not valid.

This means that the role of the Controller in the mesh is indeterminate.

This is most often seen on a cluster where when issuing an etcdctl ls /stackengine/leadership-ring there are more than 3 keys.

[destari]

Can we use atomics or compare-and-swap functions of etcd to determine atomically the state of a type of node in the mesh?