It works the same as resistances in eve AFAIK

This is the easy way to visualize it:

If a ship has zero mods on it, it likely has %0 resistance to EM damage.

If you add a %50 em hardener, you have a %50 resistance to EM damage. (all percents expressed as decimals)

So the resistance equation for the first hardner is:

EM1 = 0.50

If you add a second %50 em hardener, it affects the remaining difference in the resistance profile, but it also has a stacking penalty.

So it has 0.87 effect on the remaining %50 (since half the resistance is already taken up, it is %50 as effective). So the increased resistance equation is:

EM2 = 0.50*0.87*(1-EM1) = 0.2175

Again, a third %50 em hardner affects the remaining difference, but also the stackign penalty so

EM3 = 0.50*0.57*(1-(EM1+EM2) = ~0.08

The total resistance is expressed as EM1+EM2+EMn where n is any number of EM hardners. So our total resistance in this example would be:

EM1+EM2+EM3
=0.5+0.2175+0.08
=0.7975

Check your answer: EFT says this resistance profile should be %79.8

This has been asked so many times and people always come in and do the match according to the calculations for stacking penalties in EVE, its great that so many people are helping by explaining these but CCP should really come out and say if there is a stacking penalty on these mods and weather or not passive skill bonus's are affected by it and in what order the penalties are calculated.