Suppose that there are two populations A and B in the environment. Each population tries to evolve in order to increase its fitness which is coupled to the behavior of the other population, e.g. the performance of hunter is highly dependent on the performance of the prey. However, when A increases its fitness, B evolves in order to become fitter and then the fitness of A is not the same as before as the fitness landscape of it is highly dependent on the others’ policy. This (or something similar to this) is called Red Queen effect in co-evolutionary systems. There are many arguments about its definition and even the usefulness to consider it which you may read here.
I am not expert in co-evolution and I am not aware of the existence of the analyses regarding the dynamics of the co-evolutionary mechanisms (of course, there must be!); however, when I thought about Red Queen effect, I found this idea interesting:
We may analyze the co-evolutionary mechanism in the game theoretic framework. Red Queen effect is like not becoming stable to the Nash equilibrium in the game theory. As we do not know the model of the world (Payoff matrices and stochastic game transition probabilities) in a co-evolutionary setting, we may take a look at similar work in multi-agent reinforcement learning literature and get some inspiration. If we define rational and convergent properties as Bowling defined, we may say that the common implementation of co-evolutionary mechanism that leads to Red Queen effect is like using simple Q-learning (rational/non-convergent) in multi-agent learning. Not let we use something like Minimax-Q or Variable Learning rate Q-learning for co-evolutionary process in order to suppress the Red Queen effect.
Moreover, looking from another perspective (again game theoretic): An agent with Pure strategy may not have a Nash equilibrium but if it follows a Mixed strategy, it has. An individual in a population is certainly doing a pure strategy in most cases. What about evolving a set of strategies choosing randomly between them (evolving PDF too) to imitate evolving to a mixed strategy?!
Hmmm … don’t kill me if it is far from reality! I use my weblog exactly for this kind of conversations! Any idea about them or citing similar ideas? (I searched a little and found that R. Paul Wiegand have done some researches in this area. He used game theory to analyze the problem. I must see what he did.).