As long as the whole of the structure, including all foundations and gutters, drainpipes etc are on your side of the vertical line delineating the boundary you can build. You have no right of access at all or right to disturb the neighbours land while doing so.