Debt collectors harassing you about taxes you owe? Know your legal rights — and how to avoid scams.

Debt collectors harassing you about taxes you owe? Know your legal rights — and how to avoid scams.

Whether you paid taxes late, didn't file taxes at all — or filed but never paid, you may owe a big chunk of change to the IRS. As unpleasant as that sounds, delinquent taxpayers now have something else to worry about: The IRS has begun to use private debt collectors to get the money it's due. 

Plus, a recent rise in debt collection scams  the top consumer complaint to the Federal Trade Commission in 2016 — means that consumers really need to be extra careful when they get a call "from the IRS." Whereas previously you could dismiss such calls as likely scams, now they might not be.

Even so, there are strict rules on how debt collectors can contact you and what they they can and cannot say. Here are 3 steps to take if you're being harassed by debt collectors over unpaid taxes.

1. Make sure the collectors are legit

If you didn't pay 2016 taxes on time, but were up-to-date until then, any call about overdue taxes is almost certainly a scam: "No one will hear from a private collection firm unless they have unpaid tax debts going back several years and they have already heard from the IRS multiple times about this debt,” Mary Beth Murphy, commissioner of the IRS small business and self-employed division, told AARP.

Legitimate debt collectors must also identify where they are calling from. Only four private companies have IRS contracts to collect debt: CBE Group, ConServe, Performant and Pioneer. And these collectors are not authorized to take enforcement actions, according to AARP. That means they can't threaten to toss you in jail or take your house.

A big red flag is if the debt collector asks you to send them a check or wire them money. "Taxpayers won’t ... be asked to pay the private debt collectors," Consumerist notes. If a legitimate debt collector contacts you, and the debt is real, the only place you should have to send money is straight to the IRS — not a collection agency.

The debt collector may discuss a variety of payment options with you, but when you're ready to pay up, only do it online at IRS.gov or mail a check to the U.S. Treasury made out to the IRS.

2. Know your rights

Burly men with baseball bats shouldn't be showing up at your door, even if you legitimately owe debt. You still have legal rights: Debt collectors who enter into a contract with the IRS can't just do whatever they want.

Here are some of your rights under the Fair Debt Collection Practices Act:

• To demand the debt collector send you a written "validation notice" saying how much you owe and to whom within five days after they first contact you

• Not to be contacted before 8 a.m. or after 9 p.m.

• Not to be contacted at work (if you ask)

• Not to be contacted at all if you tell the collector in writing to stop

• Not to be lied to about how much you owe (or anything else)

• Not to be threatened with violence

Read your full list of rights, via the Federal Trade Commission, here

If you are being harassed by a debt collector, file a complaint with the FTC, the Consumer Financial Protection Bureau and your state's attorney general. You could also sue the collector and potentially collect either $1,000 or actual provable damages you suffered due to the collector's actions. 

3. Try to work things out with the IRS before you hear from a debt collector

Don't want to deal with a debt collector? Find a way to work things out before they contact you in the first place.

The IRS generally has 10 years to collect unpaid tax debts before the statute of limitations makes the debt uncollectible. If more than a decade has passed, you don't have to pay and the collector shouldn't be trying to collect.

If the debt you owe is less than 10 years old, you have a few options. You could actually enter into a payment plan — as long as you owe less than $50,000 and have filed all your required tax returns. 

You can also apply for an Offer in Compromise (OIC) which "settles a taxpayer's tax liabilities for less than the full amount owed," according to the IRS. These offers are decided on a case-by-case basis depending on your income, ability to pay and a few other factors.

Play your cards right and you'll never get stuck with an unwanted call from a dreaded debt collector.

Sign up for the Payoff: your weekly crash course on how to live your best financial life. Additionally, for all your burning money questions, check out Mic's creditsavingscareerinvesting and health care hubs for more information — that pays off.