Legislators in Washington, DC are taking a closer look at racial bias in health care algorithms after an October analysis found racial bias in a commonly used health tool. Sens. Cory Booker (D-NJ) and Ron Wyden (D-OR) released letters on Tuesday calling for federal agencies and major health companies to describe how they’re monitoring the medical algorithms they use every day for signs of racial bias.

“Unfortunately, both the people who design these complex systems, and the massive sets of data that are used, have many historical and human biases built in,” they wrote to the Centers for Medicare and Medicaid Services (CMS). The senators’ focus for their letters was mostly on gathering information. They wanted to know if the CMS, which administers Medicare and Medicaid, is collecting information from health care organizations on their use of algorithms. They asked the Federal Trade Commission if it was investigating harm caused by discriminatory algorithms on consumers and asked the health companies, including Aetna and Blue Cross Blue Shield, if and how they audit the algorithms they use for bias.

“These practices bubbling up are more like best efforts.”

Equity in health care depends on identifying and rooting out bias in algorithms. But because the programs are still relatively new, there still aren’t best practices for how to do so. “These practices bubbling up are more like best efforts,” says Nicol Turner-Lee, a fellow in the Center for Technology Innovation at the Brookings Institution.

There are tools designed to detect bias in algorithms, as the senators’ note in their missives, built by private companies like IBM and nonprofits like The Alan Turing Institute. But they’re not perfect solutions, and they don’t weed out all of the potential sources of bias. Running algorithms through their checklists should be done routinely, says Brian Powers, an internal medicine physician at Brigham and Women’s Hospital and author on the October racial bias paper. “But that won’t get you all the way there.”

The October study, for example, identified a bias that wouldn’t be found by an automated detection system, Powers says. The algorithm it analyzed is designed to flag patients with complex health care needs, and it uses the cost of care as a proxy for how sick they are. The algorithm itself was unbiased, but using cost prediction as a way to flag patients was biased because white patients are given costlier treatments than black patients. Just looking at the algorithm alone wouldn’t have identified the issue. “What we uncovered wasn’t in the algorithm and its ability to do what it was trained to do, but in the implementation, and the choice to use cost as a proxy,” Powers says.

Equity in health care depends on identifying and rooting out bias in algorithms

Bias identification tools can help to make algorithms and artificial intelligence tools more accurate, but they don’t necessarily tackle the root causes of bias, which are larger systemic issues and inequities. An open-source tool might look for problems with an algorithm, but it would not be able to identify issues with the way it was constructed, Turner-Lee says. “The biggest challenge we have with bias in AI and machine learning systems has to do with the fact that we continue to divorce them from larger structural issues that are happening in areas like health care,” she says.

Fixing the problem isn’t straightforward. While quantitative tools are necessary, they’re not sufficient, Powers says. “I think it’s going to have to be a little more nuanced.” It might be possible to generate a list of key questions, but there’s no single solution that could apply to any algorithm. Algorithms that use proxy measures, for example — like health costs as a measure of sickness — need to be examined more carefully, he says, and any bias in the proxy would have to be evaluated separately.

Removing bias will require a closer partnership between users, like health care systems and doctors and the groups developing algorithms. There should also be more attention paid to potential bias when a program is built, not just after it’s available for use in hospitals. “Companies are more reactive to problems, versus being more proactive,” Turner-Lee says.

The racial bias study from October drew attention to the issue of bias in algorithms, as will the letters from Sens. Booker and Wyden. Turner-Lee says that attention is encouraging, but there still needs to be more recognition of how structural, social issues affect technology in key areas like health care. “We’re not as close as we could be,” she says, “because we still see the technical space as being only about technology.”


Please enter your comment!
Please enter your name here