what about custom abilities, models, etc? is all this possible or hardcoded?
This is where the modding part shines with this engine.
That said, it's not a trivial effort to make a full scale mod in Sins of a Solar Empire.
You can plan to spend 2, 3 or more years if your working on multiple factions whether it is a Total Conversion or Expansion Mod.
Check moddb for some different examples. BTW all of these projects have been in development for multiple years. Mine started in 2010 for example. Several others are older than that.
Sins of the Fallen/Fall of Kobol (mine)
Sins of a Galactic Empire
Maelstrom (under Diplomacy but supports all versions of Sins).
Sins of the Prophets (beta release of UNSC only).
Star Wars Requiem
I personally think the biggest problem people have modding is they underestimate the amount of time needed after modeling and texturing. For a full faction your looking at preparing ~40-50 models(hard points, materials, tangents, mesh conversions), designing and coding ~120 research subjects, desiging and coding ~35 unique abilities, which all need various icons, pictures, buttons, etc wired to them. This also all precludes taking the time to learn XSI and the syntax structure of Sins in order to make these changes. I'd plan on atleast a 1000 hours worth of work per faction (excluding modeling/texturing).