The first time a noble is sworn, keep track of the faction that swore
him away. If that faction quits the game, the noble goes away.
That prevents you from gaining nobles via swear/quit, but permits you
to keep nobles that you capture by other means.
-- Scott T.