Most accidents occur because you don't expect them, so using your judgement on wether or not it's safe isn't an option. It's like people who think they have the right to speed because they know the road is quiet. But do they know about little Johnny, who's ball has just run into the road? Do they know they'll soon be picking pieces of his scalp out of the windscreen? No. Same goes for eating, smoking, reading, or getting oral sex from your partner whilst at the wheel. Mobile phones are getting the flak because it's another stupid thing people like to do whilst driving, and we're too tolerant on dumb ass driving already.
Simply put. If you'd fail your test for it, you shouldn't be allowed to do it. Spot fine, public flogging, and a ban from using a cell phone for a year.