I assume you can't bring out your building hammer to demolish the walls either. If you can still attack, you could try taking down the structures that way?