Why didn't you just call the police immedieatly when the delivery guy drove up on your lawn? That's criminal damage to property. The fact that he then tried to challenge you to a fight was assault, and he could have been arrested on the spot for that. In fact, you could call police now and have him arrested. Your wife was a witness. It would also make the driver more liable (and likely) to pay for the damage to the lawn.

There's no way you should have even thought about accepting delivery, or discussed paying for it.