Hey, @Fritz - from what others have said, it should come with time. It just may take a very long time depending on how much drawing you actually do; frequency and longevity. Drawing circles (and straight lines, etc) has always been difficult for me and I've decided to accept that I will never be able to draw a perfect circle, or at least as perfect of a circle (or ellipse) as I'd like it to look. Therefore, I use templates and a French curve when I can.
As a side note - one thing I still have a tough time discarding is the filter we sometimes have when looking at other artists' work, especially really good artists that we look up to. Take Jake Parker for instance. The man has a knack for creating really cool stuff. But, if you break it down and start looking at the individual pieces, the ink work is by no means "perfect". His straight lines aren't straight, his circles and curves are wobbly and so on. And I don't say this to be rude or discount his work and style - I love it. Things are in perspective, it's well designed and composed with variations in line weight and so on. We just glaze over the finer points and look at the whole because we didn't draw it ourselves. When you create something, you nit pick all the little details. You sweat over making that circle perfect or that line straight because you are constructing it.
Just my long-winded opinion/perspective, but hope that helps some.