My understanding is that if a horse causes damage, the owner is liable. It is no defence for him to claim he was not negligent ... or even that a field gate was deliberately left open so that the horse could escape. If you have a horse, you are liable, in that you did not ensure the horse could not cause damage. I'll see what I can find.
In the meantime, if the rider was not negligent, as his insurers claim, then he must have done it deliberately, or he was incompetent. Are there any other options? Of course, you don't know what the rider told his insurers, but at least his insurers have agreed that a specific horse, with a specific rider, was involved in a specific accident.
Imagine if the hand brake had failed on your car, and it had rolled into the horse. You were not negligent, you had applied the hand brake. The hand-brake had never failed before. You can show that the car is regularly serviced. How do you think tis rider would react, if you offered the same defence as he has to you.
My first thought was to make a Small Claim (£25?). My second thought was that a hunt follower may have "friends in high places" ... if such a thing was possible. I don't know how you get around that. ???
Do you have Legal Assistance on your car insurance? For about £20 a year, it beats paying for your barrister, while the other party gets theirs paid for by their insurers. You could ask your insurers to pay for your repairs ... and then chase the rider's insurer's to recoup their losses.