Lord Pagan is a old knight who lived on the outskirts of Pubton (At least several days travel), who claimed Pubton owed him taxes (Due to technically being on his lands). He has a small army's worth of slaves who work for him, as well as maintain his property (Although they're treated fairly well and might not be entirely considered slaves).
It was during a attack by the Non and Kierkegaard that Lord Pagan's own manor was taken from him. Causing him and Dragomir to travel from Pubton to deal with the threat. Which then led to Pagan re-locating his manor into Pubton itself as both a way to protect himself and his servants, and to collect taxes/strengthen Pubton with his presence.
It should also be noted that during a confrontation with Oswald the Farmer (Dragomir's Father) that Pagan sliced off Oswald's remaining arm, and they've been best friends since (Not the most awkward pair in the Dragomir universe mind you).