It really depends on your shooting style and why you are going for Primes.
Before buying the 12-40mm, I tended to use the Panasonic 20mm f/1.7 mark I a lot, and then also the Olympus 45mm f/1.8. In general, the 12-40mm is one of my trinity of go-to lenses (the Olympus 14-150mm f/4-5.6 mark II for general outdoors shots and the Panasonic 35-100mm f/2.8 for longer range low light shots). The 12-40mm replaced a lot of the uses of those two primes, and in normal low light setups, it is sufficient (and 1-2 stops faster than the consumer zoom lenses, depending on whether you are at the wide angle or telephoto ends).
But there are times when I find going back to the primes, because at times I am shooting in situations where I need that extra stop or so of light.
Sometimes, I just put on a prime onto a small body and it doesn't call attention to it like the 12-40mm on the larger body.
Some people prefer the primes, because it forces them to compose the picture for the given focal length, and having to do that forces you to actually think about what you want the final shot to be. But there are other times, when it is just impractical to zoom with your feet to get the shot you want.
But the 12-40mm and other lenses are expensive compared to the consumer lenses. In the end of the day, you have to choose how much you want to throw at buying new lenses, and what the return on investment is. And there is always the temptation of getting the 'next' lens and thinking it will solve your problems.
And in the Judeo-Christian framework, I tend to think the 10th commandment (though shalt not be envious) is the hardest commandment to keep, and forums like these tend to add to the GAS (gear aquisition syndrome).
If you have never shot with primes before, do an experiment. Set your focal length to be the prime you wish to buy (or at least close enough). Spend some time shooting only at that focal length. See whether the prime would work with what you want to shoot. For some it does, and others it does not. Also keep a log of the number of times when your current lens would not let get the shot you want, and determine whether you want something wider, something longer, or something with a faster aperture.
If you have a weather sealed body, keep track of how often you shoot in the rain. Note, that most of the cheaper primes are not weather sealed.
Remember, everything is a trade-off. Sometimes you have to give something up to get some other benefit.
