The signature gatherers I’ve met have essentially been day laborers. If their hustling seems desperate, that’s because they’re doing piecework. They aren’t enthusiastic foes of liberty, whatever you might think of the referendum they’re working on or ballot questions in general.
I assume some other signature gatherers are genuine activists.
In either case, use courtesy. Save the soapboxing for a time when it has a macroscopic chance of success. Keep the blistering denunciations to yourself, too — the day laborer is probably depressed enough already, and the zealot will just feel more righteous for enduring your “persecution.”
Besides, a simple “no” wastes less of your time than “no, I don’t believe in referendum questns, the political process is inherently corrupt, and the question you’re pushing is especially terrible for the following list of reasons, how could you even imagine putting that on a ballot, you must be an utterly brainless garbage subhuman.”