I have the same issue (I expect most do). Not really with Velcro pants, but with the Velcro on my laptop bag. It tends to tear up the passenger seat.
In a few years, I will probably just have it reupholstered. Only costs about $40 per seat if you supply the material.