Are you worried about getting swindled out of your savings or losing everything in a crash? Alice Finn, the author of Smart Women Love Money, explains the simple ways to avoid those catastrophes—and warns us about a threat to our earnings one never seems to consider.

If I stopped a random woman on the street and asked her what she thought the biggest controllable risk to her long-term portfolio potential was, I can almost guarantee she wouldn’t guess the right answer. She might say it was the risk of being invested during a big market downturn, like the bursting of the dot-com bubble or the great recession of 2008. She might mention the possibility of being taken for a ride by a Ponzi schemer like Bernie Madoff, or investing in a fraudulent company or one that goes bankrupt. Perhaps she’d mention the risk of not investing at all—sitting on the sidelines and watching as the market goes up without her taking advantage and growing her assets.

Yes, market declines come and go, but if you remain invested in a diversified portfolio of low-cost index funds and regularly rebalance, you ensure you’ll always be in a position to profit from the upturns when they come. If you’re sticking to index funds, there is no risk you’ll end up in the clutches of a Bernie Madoff and his “proprietary” (in this case, downright illegal) investment strategies and products, because you’ll be buying plain vanilla funds that trade in the public markets. And of course, you’ll be staying invested at all times, because that’s the key to making money at all. As long as you invest, and stay invested, your returns keep compounding. The more time you have, the more you can reinvest those earnings and leverage the potential of your original assets.

But there is another risk, and it’s one that almost no one ever seems to consider. The one most of us forget about until it’s too late. It doesn’t grab headlines in the same way a market crash or a renegade money manager might. But it can wreak havoc on how much money you end up with when you have to look at how your nest egg will provide for you and your family.