The answer is both. Since your filament diameter varies so much, you need to adjust your extruder to be able handle it.
Increase tension in the idler so it can press on the thinner filament. Be sure also to put the average size out of at least 10 readings in the slicer.