Both Trudeau and Mulroney simply became normal - if very upper-crust - citizens once no longer in office, and did enjoy walking about on the streets, not "downcast" at all.
Amusingly, I can put your comment to the test. A few years ago my wife and I were walking along rue de la Montagne (downtown Montreal) as a long black limo came to a stop beside us. Before I knew what was happening my wife called out, "You were the worst Prime Minister we ever had!" (This was before Harper had connived his way into power) as Brian Mulroney got into the limo. We kept walking, but an intuition made me turn round, and sure enough, Mulroney opened the door, got back out of the limo, and somewhat perplexingly said, "Yeah?! Well with people like you around, who can you blame?!". Then he got back into the limo.
As for Trudeau... I live a block from his house, and saw him often as he walked home from work at Heenan Blaikie. I never witnessed any incident to put him to the test, but (although I have no love of the Martial Law-imposing, pro-business power-tripper) I can't really see him rising to the bait so lamely as Mulroney did.