Sfdwhf

Consider a population where a low percentage of people (5%) posess a genetic advantage over the rest. In my fictional world, let’s assume this small percent of the population can wield magic while others cannot.

Now over the course of millenia, the count of the advantageous group should increase due to natural selection and eliminate the larger less-advantageous group.

I would like to know how this can be prevented from happening - the possible exceptions that cause this minority to still remain a minority without going extinct.

Infertility, magic talent comes with a severe reduction in fertility if not down right sterility. Now would that not make them die out? Not of the mutation rate among regular humans is high enough.

So your regular humans every now and again have a child with magical ability but it's sterile. So the child will never pass on their highly effective magical genes.

Species that are as complicated as humans typically take many millennia to evolve, but the 2 basic approaches are.

Sexual Selection

If you don't want a gene to reproduce, simply make it undesirable for the population to breed with those of "genetic advantage". The reasons for such could be cultural (anti-magical bigotry), biological (humans at the time see the trait for magic produces ugly people), or purely functional (maybe those with magic cause unintended spells that occasionally slay offspring).

Recessive Trait

Make the gene for Magic be a recessive one. In the same way that blue eyes are more rare than brown ones, magic genes may be rarer that non magic genes.

Sickle cell anaemia and malaria

In a malaria-ridden country, sickle cells provide a marginal advantage to survival because they give immunity to to the disease despite causing some ill-health. The condition allows humans to reach reproductive age.

In malaria-free countries sickle cell is a distinct disadvantage.

A magical ability came about by a genetic mutation. Unfortunately that mutation either reduces fertility or makes for a high death rate before puberty.

The following is a fairly technical explanation but easier ones can be found in more popular-type articles.

Heterozygotes for the sickle cell gene are relatively protected
against malaria, while patients who are homozygous for the sickle cell
gene, suffer from sickle cell disease and are highly prone to the
lethal effects of malaria.
http://www.scientificanimations.com/malaria-loses-sickle-cell-battle/

Well there are a few ways this can be achieved.

They may have worse health overall. I don't know what sort of magic your users have access to, but maybe they're more susceptable to diseases or just live shorter than magicless humans.

Another angle could be to give them lower fertility that magicless humans.

The magicless humans keep the population down. It's not hard to figure out that a guy who has access to magic might be a serious problem for you if he or she decides he or she doesn't like you. The solution? Ensure that there aren't enough of them to be a threat to you.

Basic exceptions to this rule of nature are A)lack of competition, B)separation, C)natural disasters, and D)Human interference.

Feel free not to read this explanation if you don't think you need it:
If I'm not mistaken, Natural Selection is the idea that organisms more fit for the environment are more likely to survive than those that are not as well designed for the environment. If this is the case there are a few elements that affect this process. We can find out what these are by thinking about what can make things more fit for a certain environment, or what things create opportunity for superiority. Right off the bat, we have competition. the more things in an environment the more apparent Natural selection is. If there is very little in the way of competition many things escape natural selection. For example, a pack of wolves kills a deer, they all eat equally at first, but one day a wolf is born with a mutation in its genes that makes its teeth sharper. this wolf has a competitive edge and will tear more meat off the animal than the others because it can rip and chew faster and more effectively. Thus he would be healthier and become the alpha and have more children than the other males. Eventually this one mutation would overrun the wolf population, natural selection right? but an exception to this is lack of competition, if there is enough deer that all the wolves eat right then sharper teeth have no place in the world of natural selection, however longer or stronger legs might. moving on to another exception to natural selection is separation, say there are two islands with extremely similar environments, and both have wolves on them. both islands have plenty of deer so no wolves have to compete for food. however on island one the wolves have stronger legs. they catch more deer, but natural selection never takes place and both wolf races live on equally because they are separated. Another exception to this process is natural disasters, such as a volcano erupting. this kind of thing could kill off a developing superior mutation and completely stop Natural Selection in its tracks. And lastly, the biggest exception to Natural Selection is human interaction and interference, every time you see a video of somebody saving an animal from an icy pond or something that is an example of humans interfering with Natural Selection. The trapped animal did something that made it fail at life, for whatever reason. this means the animal wasn't entirely ideal for the environment. As nature would have it this animal should have died. but humans don't work like that i guess. I hope this answer was helpful and thank you for your patience as you read this long and drawn out and probably too late response.