I have never encountered that issue so far.

Often times I find there is more than one way to get down/up from lower levels like you are facing.

Maybe another path behind a wall you can break down?