Here's how much an early 401(k) or IRA withdrawal really hurts you once you retire

If you have been diligently saving for retirement, by your early thirties you should start to have a nice little nest egg — about the equivalent of your annual salary if you're really on track. Even if you've been less careful, hopefully you have saved up tens of thousands of dollars. 

All that money can start to become a pretty big temptation if you're not careful. After all, it's your money, isn't it? Who is to stop you from dipping your finger into the well if you need a little loan?

And what's a couple thousand dollars in the long run? Odds are, a lot more than you'd think.

If you borrow just $5,000 from your 401(k) plan over the course of five years, you actually cost yourself closer to $7,000, according to Bankrate's retirement account calculator. And that's if you pay back the loan on time. If you don't, then the effect of borrowing the cash balloons.

Indeed, your total losses on that same loan could run as high as nearly $189,000. That's through a combination of taxes, the 10% penalty on early 401(k) or Individual Retirement Account withdrawals, and the gains that $5,000 would have made it if it had been sitting in your account for the 35 years between the loan and retirement.

That's a reason to not take your nest egg — however large — for granted.

Yet despite the potentially enormous opportunity cost, people still do borrow from their retirement accounts; a new Wall Street Journal report says companies are increasingly anxious that this practice will keep employees from ever being able to retire and make way for cheaper young hires.

One in five retirees with access to a 401(k) take a loan from the account, according to the report, and that has the effect of reducing retirement wealth in the United States by about 25% over 30 years.

Now, in some cases, these loans make sense. For one, they're a lot more convenient than applying for a personal loan, and you're often paying interest to yourself as opposed to some other lender, like a credit card company.

The IRS also permits a few exceptions to the 10% penalty on early IRA or 401(k) withdrawals: if you are older than 59 and a half, if you are disabled, if you are a qualified military reservist called to active duty, and if you have medical expenses costing more than 10% of your adjusted gross income. People with IRAs — though not 401(k)s — can also withdraw early without a penalty for qualified education expenses, health insurance premiums paid while unemployed, and if you are buying a home for the first time (up to $10,000).

In general, if you really need the cash, you'll definitely be able to pay it back, and after doing the math you find it's the lowest cost loan available, then borrowing from your retirement account might make sense, says Bankrate. 

Just be vigilant about your other financial behaviors: The problem is, taking a loan from your 401(k) seems to accompany other irresponsible moves that eat into earnings as well. One Fidelity study found that within five years of taking out a loan, 40% of savers reduced their contribution rate, and a third of those borrowers stopped saving for retirement altogether.

Finally, to avoid being in the situation where you need to make an early withdrawal in the first place, in addition to saving for retirement you should also have three to six months of expenses in an emergency fund.

That way you'll never have to worry about taking your retirement savings off track. Here's a guide to the five types of accounts every adult should have.

Sign up for The Payoff — your weekly crash course on how to live your best financial life. Additionally, for all your burning money questions, check out Mic's credit, savings,career, investing and health care hubs for more information — that pays off.