I used not to eat nuts in a similar way (in my case I thought at the time they were causing stomach problems). I used the formulation "I don't eat large quantities of nuts" with a reasonable amount of success.