Friday, February 21, 2025, 12:09PM |  21°
MENU
SECTIONS
OTHER
CLASSIFIEDS
CONTACT US / FAQ
Advertisement
A sign marks an entrance to the Robert F. Kennedy Department of Justice Building in Washington, Monday, Jan. 23, 2023. The Justice Department has been scrutinizing a controversial artificial intelligence tool used by a Pittsburgh-area child protective services agency following concerns that it could result in discrimination against families with disabilities, The Associated Press has learned.
1
MORE

Allegheny County child welfare algorithm faces Justice Department scrutiny

Patrick Semansky/AP

Allegheny County child welfare algorithm faces Justice Department scrutiny

The Justice Department has been scrutinizing a controversial artificial intelligence tool used by the Allegheny County child protective services agency following concerns that it could result in discrimination against families with disabilities, The Associated Press has learned.

The interest from federal civil rights attorneys comes after an AP investigation revealed potential bias and transparency issues about the opaque algorithm that is designed to assess a family’s risk level when they are reported for child welfare concerns in the county.

Several civil rights complaints were filed in the fall about the Allegheny Family Screening Tool, which is used to help social workers decide which families to investigate, AP has learned.

Advertisement

Two sources said that attorneys in the Justice Department’s Civil Rights Division cited the AP investigation when urging them to submit formal complaints detailing their concerns about how the algorithm could harden bias against people with disabilities, including families with mental health issues.

Depicted here in a photo illustration, a predictive risk modeling tool used by the Allegheny County Department of Human Services to help to decide which allegations of child neglect warrant investigation has helped to improve outcomes and reduce bias, according to an evaluation released by the county last week.
Kate Giammarise
Can an algorithm help keep kids safe? So far, Allegheny County's screening tool is improving accuracy

A third person told AP that the same group of federal civil rights attorneys also spoke with them in November as part of a broad conversation about how algorithmic tools could potentially exacerbate disparities, including for people with disabilities. That conversation explored the design and construction of Allegheny’s influential algorithm, though the full scope of the Justice Department's interest is unknown.

All three sources spoke to AP on the condition of anonymity, saying the Justice Department asked them not to discuss the confidential conversations, and two said they also feared professional retaliation.

Wyn Hornbuckle, a Justice Department spokesman, declined to comment.

Advertisement

Algorithms use pools of information to turn data points into predictions, whether that’s for online shopping, identifying crime hot spots or hiring workers. Many child welfare agencies in the U.S. are considering adopting such tools as part of their work with children and families.

Though there’s been widespread debate over the moral consequences of using artificial intelligence in child protective services, the Justice Department’s interest in the Allegheny algorithm marks a significant turn toward possible legal implications.

Supporters see algorithms as a promising way to make a strained child protective services system both more thorough and efficient, saying child welfare officials should use all tools at their disposal to make sure children aren’t maltreated. But critics worry that including data points collected largely from people who are poor can automate discrimination against families based on race, income, disabilities or other external characteristics.

Robin Frank, a veteran family law attorney in Pittsburgh and vocal critic of the Allegheny algorithm, said she also filed a complaint with the Justice Department in October on behalf of a client with an intellectual disability who is fighting to get his daughter back from foster care. The AP obtained a copy of the complaint, which raised concerns about how the Allegheny Family Screening Tool assesses a family’s risk.

Elaine Jans, left, who works at the Lawrenceville Family Care Connection, serves food at the annual Family Support Center picnic at the Monroeville Community Park in Monroeville on Friday.
Kate Giammarise
With 'Hello Baby' officials aim to reach all babies born in Allegheny County -- with an algorithm targeting those most at risk

“I think it’s important for people to be aware of what their rights are, and to the extent that we don’t have a lot of information when there seemingly are valid questions about the algorithm, it’s important to have some oversight,” Ms. Frank said.

Mark Bertolet, spokesman for the Allegheny County Department of Human Services, said by email that the agency had not heard from the Justice Department and declined interview requests.

“We are not aware of any concerns about the inclusion of these variables from research groups’ past evaluation or community feedback on the (Allegheny Family Screening Tool),” the county said, describing previous studies and outreach regarding the tool.

Allegheny County said its algorithm has used data points tied to disabilities in children, parents and other members of local households because they can help predict the risk that a child will be removed from their home after a maltreatment report. The county added that it has updated its algorithm several times and has sometimes removed disabilities-related data points.

The Allegheny Family Screening Tool was specifically designed to predict the risk that a child will be placed in foster care in the two years after the family is investigated. It has used a trove of detailed personal data collected from child welfare history, as well as birth, Medicaid, substance abuse, mental health, jail and probation records, among other government data sets. When the algorithm calculates a risk score of 1 to 20, the higher the number, the greater the risk. The risk score alone doesn’t determine what happens in the case.

The AP first revealed racial bias and transparency concerns in a story last April that focused on the Allegheny tool and how its statistical calculations help social workers decide which families should be investigated for neglect – a nuanced term that can include everything from inadequate housing to poor hygiene, but is a different category from physical or sexual abuse, which is investigated separately in Pennsylvania and is not subject to the algorithm.

A child welfare investigation can result in vulnerable families receiving more support and services, but it can also lead to the removal of children for foster care and ultimately, the termination of parental rights.

The county has said that hotline workers determine what happens with a family’s case and can always override the tool’s recommendations. It has also underscored that the tool is only applied to the beginning of a family’s potential involvement with the child welfare process. A different social worker who later conducts the investigations, as well as families and their attorneys, aren’t allowed to know the scores.

Allegheny’s algorithm, in use since 2016, has at times drawn from data related to Supplemental Security Income, a Social Security Administration program that provides monthly payments to adults and children with a disability; as well as diagnoses for mental, behavioral and neurodevelopmental disorders, including schizophrenia or mood disorders, AP found.

The county said that when the disabilities data is included, it “is predictive of the outcomes” and “it should come as no surprise that parents with disabilities … may also have a need for additional supports and services.” The county added that there are other risk assessment programs that use data about mental health and other conditions that may affect a parent’s ability to care for a child.

The AP obtained records showing hundreds of specific variables that are used to calculate the risk scores for families who are reported to child protective services, including the public data that powers the Allegheny algorithm and similar tools deployed in child welfare systems elsewhere in the U.S.

The AP’s analysis of Allegheny’s algorithm and those inspired by it in Los Angeles County, Calif., Douglas County, Colo., and in Oregon reveals a range of controversial data points that have measured people with low incomes and other disadvantaged demographics, at times measuring families on race, ZIP code, disabilities and their use of public welfare benefits.

Since the AP’s investigation published, Oregon dropped its algorithm due to racial equity concerns and the White House Office of Science and Technology Policy emphasized that parents and social workers needed more transparency about how government agencies were deploying algorithms as part of the nation’s first “AI Bill of Rights.”

The Justice Department has shown a broad interest in investigating algorithms in recent years, said Christy Lopez, a Georgetown University law professor who previously led some of the Justice Department’s civil rights division litigation and investigations.

In a keynote about a year ago, Assistant Attorney General Kristen Clarke warned that AI technologies had “serious implications for the rights of people with disabilities,” and her division more recently issued guidance to employers saying using AI tools in hiring could violate the Americans with Disabilities Act.

“They are doing their jobs as civil rights investigators to get to the bottom of what’s going on,” Ms. Lopez said of the Justice Department scrutiny of Allegheny's tool. “It appears to me that this is a priority for the division, investigating the extent to which algorithms are perpetuating discriminatory practices.”

Traci LaLiberte, a University of Minnesota expert on child welfare and disabilities, said the Justice Department's inquiry stood out to her, as federal authorities have largely deferred to local child welfare agencies.

“The Department of Justice is pretty far afield from child welfare,” Ms. LaLiberte said. “It really has to rise to the level of pretty significant concern to dedicate time and get involved.”

Emily Putnam-Hornstein and Rhema Vaithianathan, the two developers of Allegheny’s algorithm and other tools like it, deferred to Allegheny County’s answers about the algorithm’s inner workings. They said in an email that they were unaware of any Justice Department scrutiny relating to the algorithm.

Researchers and community members have long raised concerns that some of the data powering child welfare algorithms may heighten historical biases against marginalized people within children protective services. That includes parents with disabilities, a community that is a protected class under federal civil rights law.

The Americans with Disabilities Act prohibits discrimination on the basis of disability, which can include a wide spectrum of conditions, from diabetes, cancer and hearing loss to intellectual disabilities and mental and behavioral health diagnosis like ADHD, depression and schizophrenia.

Ms. LaLiberte has published research detailing how parents with disabilities are disproportionately affected by the child welfare system. She challenged the idea of using data points related to disabilities in any algorithm because, she said, that assesses characteristics people can’t change, rather than their behavior.

“If it isn’t part of the behavior, then having it in the (algorithm) biases it,” Ms. LaLiberte said.

First Published: January 31, 2023, 3:55 p.m.
Updated: February 1, 2023, 1:01 p.m.

RELATED
Auditor General Eugene DePasquale
Kate Giammarise
Auditor praises Allegheny County's use of algorithm in child welfare screening
SHOW COMMENTS (2)  
Join the Conversation
Commenting policy | How to Report Abuse
If you would like your comment to be considered for a published letter to the editor, please send it to letters@post-gazette.com. Letters must be under 250 words and may be edited for length and clarity.
Partners
Advertisement
Longtime KDKA-TV host Jon Burnett on May 22, 2019.
1
a&e
Jon Burnett, long a KDKA-TV staple, leaves legacy of ‘putting good out into the world’
The Breezewood Interchange is dotted with gas stations, chain restaurants and souvenir shops. The Pennsylvania Turnpike Commission has selected Chicago-based Alfred Benesch and Company to spearhead a major redesign of the notorious roadway.
2
business
So long, Breezewood: Chicago firm selected to redesign infamous Pa. Turnpike interchange
Commerce Secretary nominee Howard Lutnick speaks after President Donald Trump signed an executive order in the Oval Office on Feb. 13.
3
news
Trump expected to take control of USPS, fire postal board, officials say
Saraswati Adhikari and her family, not pictured, work with attorneys Jillian Lukac, Elizabeth Tuttle, and Larry Lebowitz to fill out paperwork to become United States citizens during a volunteer event organized by Jewish Family and Community Services Immigration Legal Services and the Bhutanese Community Association of Pittsburgh, Saturday, Feb. 23, 2019, at the Whitehall Public Library in Whitehall.
4
news
Allegheny County will kick in $224,000 for refugee settlement services after federal funds are cut off
Police investigate the scene of a double shooting in Pittsburgh's Garfield neighborhood on Thursday, Feb. 20, 2025.
5
news
Suspect at-large after 2 shot and killed, toddler unharmed in car in Garfield
A sign marks an entrance to the Robert F. Kennedy Department of Justice Building in Washington, Monday, Jan. 23, 2023. The Justice Department has been scrutinizing a controversial artificial intelligence tool used by a Pittsburgh-area child protective services agency following concerns that it could result in discrimination against families with disabilities, The Associated Press has learned.  (Patrick Semansky/AP)
Patrick Semansky/AP
Advertisement
LATEST news
Advertisement
TOP
Email a Story