The amount of Ergot tainted grain you would need to consume to have hallucinations would mean you would be very sick. It wouldn't be an otherwise healthy person hallucinating, but a very ill delirious person.