We all know we’re not supposed to shop at H&M. Or Zara. Or Forever 21. Or any other fast-fashion retailer. Headlines of workplace abuse, factory accidents, and child labor have abounded since the mid-’90s. The new society-wide consciousness of these workplace horrors means that we now feel guilty about shopping unethically. We know we’re not supposed to. But we still do.

Why is it so difficult to feel good about what we’re buying? Well, because it’s nearly impossible. Fashion habits have changed drastically in the past 25 years, since, as Michael Hobbes says in his 2015 Highline article The Myth of the Ethical Shopper, “the apparel industry, the entire global economy, has undergone a complete transformation. The way our clothes are made and distributed and thrown away is barely recognizable compared to the way it was done in the ’90s.” 

And sketchy labor is easy to hide. A brand like Wal-Mart might hire a mega-supplier to fulfill a clothing order, which might outsource some work to another company, which might outsource to then another. The original clothing brand often has no idea where exactly its clothing is being made. In 2012, through this chain, Walmart's production happened at a factory that the store had banned—a factory that burned down.

“Mass subcontracting, which outsources production to other factories, makes it difficult to track how and where our clothing is being made,” said Micaela Marini Higgs in her 2017 Racked article Ethical Shopping Is Nearly Impossible. “Each piece of your clothing, from the zipper on your pants to the buttons on your dress, has a different and complex supply chain.”

There are other kinds of bad ethics besides corrupt labor practices that U.S. consumers often care more about, but that further complicates our ability to shop ethically. Higgs mentions that “The Salvation Army has a history of discrimination against the LGBTQ community, and Goodwill pays its disabled employees less than minimum wage.” Ivanka Trump’s fashion line was dropped from Nordstrom for political reasons. 

I knew about the anti-gay Salvation Army and about anti-Ivanka Nordstrom. But I had never before heard about the Walmart factory fire that killed 112 people and injured hundreds. Are our ethical priorities in the wrong places?

And then there’s cost. Most people I know don’t shop ethically because they can’t exactly afford to. An ethically-made dress or shirt won’t really go for less than $50, and prices usually stretch much higher. But here’s the thing: we purchase so many clothes. Why do we have a problem spending $100 on an ethically-made dress, but no problem buying more than five $20 tee shirts in a year? Minimalism and quality over quantity are key to conscience shopping.

In a 2012 TED Talk, reporter Leslie T. Chang stressed the importance of including the voices of factory workers in the conversation about ethically-made products. “We must be peculiarly self-obsessed to imagine that we have the power to drive tens of millions of people on the other side of the world to migrate and suffer in such terrible ways,” she said. “Chinese workers are not forced into factories because of our insatiable desire for iPods. They choose to leave their homes in order to earn money, to learn new skills, and to see the world.” Prattling about in an ethical, $250 dress does nothing to help those workers. It only buys the purchaser out of guilt.

The narrative about fast fashion often revolves around Westerners and how we can shop in ways that make us feel less guilty. But the factory workers are people just like us and are being excluded from the narrative in which they play a central role. Just boycotting Zara might not fix much—as Higgs’s article points out, boycotts can harm workers when factories simply shift production to a different region.

So, what does this boil down to? That ethical shopping is difficult. Even companies that think they’re being moral might have their production outsourced to an unsafe factory. Companies can be unethical in many ways, and simply boycotting brands might not always help conditions for workers who are making better lives for themselves.

Here are my tips: educate yourself. Online resources breakdown how to best shop ethically, like the website fashionheroes.eco, and a roundup of sustainable brands, many of which belong in the under $100 range, are available instantly. (For starters, check out Everlane and Reformation.) Thrift from local consignment shops, or wear your relatives’ hand-me-downs. Learn about the outsourcing practices of your favorite brands—many of them might use factories in Asia full of young women earning a living and independence.

And don’t buy something that you won’t wear in 25 years. Make clothes last.