Let's talk about how a lord's power heavily depended on the loyalty of his vassals, which wasn't guaranteed. Vassals could, and often did, play their lords against each other. The whole system was really about these relationships and local power, not some absolute pyramid scheme.