Looks like u dont know people playing this game. Ladders would be just another griefing tool. Just go inside small pallisade, blow up house because nobody will show up, and than just leave.
I understand the concern you have is around sieging of houses. Clearly, people can already blow up a house and then leave. I don't see that changing. You're worried that with ladders the siege will begin at the house, and people will not need to take out the wall. Making sieges potentially less expensive and easier. I get it.
Again, the cost of ladders is up for consideration. The HP of houses, keeps, and other structures is also variable. It does not need to be your worst case scenario. The exact amount of HP of a typical wall could be added to a maxed house or keep, keeping your ratio that you pressumably like exactly the same. What seems right to you?
The practical implications of taking out a wall to then siege a house means more mang and boulders. In exchange for that, the attacker gets full access to the palisade until the wall is replaced. I think that's an important difference for a siege that ladders do not provide.
The benefits of ladders goes beyond griefing, though I'm sure it will be used for that, like all tools they can be used as weapons. Disruption of expansion through small scale conflict, PVP fighting, banditry, access to content behind walls, infiltration, increased risk behind walls, a check on safety through gold expenditure, all enabled with ladders that are just currently very rare in the game. A stagnant hell. Maybe you decide not to siege the TOFU house because it's more fun to climb their palisade and fight with them over resources every few days. That seems like a win for everyone in a game like MO. What about this do you not like?