Hello all,
Looking for some advice or insight.

We own a house that has a self-contained annexe with its own entrance, address, and garden access (via its own French doors). It’s connected to the main house by a shared internal hallway, with lockable doors on both sides.

We had to move abroad for work and rented out only the main house on a 3-year lease (stupidly, with no break clause). Unexpectedly, we’ve had to return after a year and were planning to move into the annexe. However, the tenant in the main house is now claiming that our presence in the annexe affects their “quiet enjoyment” and insists he has exclusive use of the garden.

We were willing to offer to let them leave the lease early and keep their deposit, but this was rejected.

We now want to put up a fence to section off the part of the garden that’s directly accessible from the annexe for our own privacy and use, but they are not agreeing to anything and refuse to communicate in person. They have also failed to maintain the garden at all, which was in great shape when we left (we'd invested thousands into it). We're considering a Section 8 for breach of contract due to the lack of maintenance, but we realise that's a long shot.

We’re speaking with our estate agents (whom we have legal cover with), but curious what others think. Has anyone dealt with a similar situation? Are we being unreasonable for wanting to move into the annexe and reclaim part of the garden?