Garden Law Discussion

My neighbour has queried the way I have erected a feather edge fence along our boundary. He says that although it is my fence and I have paid for it, the pretty side should face his garden and I should have the arris rail side and the brackets on my side. Is this correct?