The way I'd put it is that you don't have the facilities for the same kind of control over texture and happy accidents when working in vector. Everything has to be precise and mathematically representable, making it difficult to work with textures and fine, arbitrary haphazard details.
You know that touch of rough linework or weird crust in textures? Fuck you, vectors are clean and infinitely perfect. If you want rough linework in vectors, you have to describe said roughness in detail. Even if you zoom in to an absurd amount, it will always look clean.
They're infinitely scalable, always as sharp or smooth as intended. There is no set size at which it looks best, so you, as the artist, have to provide that detail. This can be tricky, as you can't just paint on top of it. That's raster stuff, with discrete detail. You need mathematically representable results. Vector programs in general don't support this kind of on-the-fly editing, and if they do, they're almost always flawed in how they describe your very specific brush strokes to an infinitely expandable texture applied to an infinitely precise shape.
Raster is exactly what it is. It's a discrete grid of discrete pixels, and what you see is what you get. You can't get more precision by zooming in closer. Then again, if that's what you get, you can just slap in arbitrary detail wherever. Got some wack-ass brush with a wack-ass texture, want to hand-render gradients, do some very subtle hue shifts or just capture your organic inaccuracies? You can do that. Do that without having to have the computer interpolate and artificially smooth out your imperfections.
Vectors might take over completely one of these days, as tools to convert user input into readable details at specific zoom levels, but right now, vectors can't do arbitrary detail too well, and always look sterile.
If you're working in something with clear linework, flat or minimally changing colorwork, vectors are good. If not, stick with raster. Up the resolution if you have to.
tl;dr vectors suck at textures and anything messy