Notice a subtle point here: Seneca isn’t saying that prosperity is not worth pursuing. It is, after all, a preferred indifferent. But it is preferred only insofar it doesn’t get in the way of conducting a virtuous life, as one gets the sense Lucilius was worrying about insofar his own pursuits were concerned. Which is why his friend reminds him that he is under no obligation at all to live in the fast lane.
Just yesterday, for the first time ever, I considered removing the rear-view mirror from the Jeep. (Instead, I twisted it upwards to view the roof.) Since the Jeep is slow and old, as am I, there is ALWAYS someone tail-gating me. I’ve narrowly avoided accidents, where watching the tail-gater behind me distracted me from the road ahead. I’m so much in the “slow lane”, I am literally being run over. Where, really, are you going?