Needle for woven and knit together?

So as I've been sewing this, I've noticed something- a needle issue. I use a ballpoint for the knit. I used a regular sharp for the woven, but when I sew woven and knit together, which should I use? The sharp will "cut" the knit fibers rather than push them out of the way, but the ballpoint on the woven sounds like the dullest needle grinding through the woven and like I am about to break something. So which do you prefer when sewing knit and woven together? Does it matter which fabric is on top? Just wondering how you all handle it.