A Faction is most commonly a liveried group of minifigs used as troops in an armed force. It can be an official subtheme or a fan developed theme. The term is used primarily within the Castle community when building medieval or fantasy themed armies but many fan created themes within the Space community also incorporate factions.
The primary difference between a fan created theme and a custom faction seems to be that custom factions are the design and property of one single creator while fan created themes go beyond simple troops and include vehicles and installations built by a number of different fans.
Although not a term in general use within The LEGO Company the idea of factions is widespread in the Castle line where heraldic designs provide various subthemes such as the Black Falcons. In a more modern theme the Alpha Team is in constant conflict with the evil Ogel organization which is also identified by uniforms based on heraldic designs.