How the penalty is calculated depends on the lender, the mortgage, and the type of rate you have.
For many closed fixed-rate mortgages, the penalty is generally the greater of three months' interest or an interest rate differential, often called an IRD.
An IRD is meant to compensate the lender for some of the interest they may lose when you pay the mortgage out early.
Variable-rate mortgage penalties are often based on three months' interest, but the exact calculation still depends on the lender and mortgage agreement.
How the lender calculates it matters a lot. Two people with similar mortgage balances and rates can end up with very different penalties.