The growth of algorithm-based automation – defined as automated workflows that perform problem-solving operations – is transforming how governments enhance inclusive community development initiatives. This technology can streamline administrative processes and improve decision-making efficiency. As automation continues to evolve, its potential to promote social equity is garnering increasing attention. However, algorithm-based automation’s short history has shown that a fair application of this technology can be challenging, as systemic biases have historically overlooked marginalized individuals. As a result, government agencies must focus on the equitable application of automation to better address disparities and enhance the delivery of public services. This article examines the benefits and challenges algorithm-based automation can create for underserved communities.
A key advantage of automation is the ability to process extensive amounts of data expeditiously and at scale. By analyzing demographic data such as income, education, and healthcare access, government agencies can identify underserved populations and target interventions more precisely. The application also allows governments to bridge gaps in service delivery and reduce disparities.
A case in point is how New York City uses algorithm-based automation to enhance city management, harnessing the technology to optimize everything from public safety to building inspections. By processing vast amounts of data daily, city officials can make informed decisions that improve efficiency and public welfare.
For instance, the NYC Fire Department (FDNY) identifies high-risk properties by analyzing historical fire data, building conditions, and other relevant factors, allowing inspectors to prioritize their efforts. Using an Oracle-based program with data-mining capabilities to better anticipate where fires may spark, FDNY relies on a tool called FireCast, an algorithm that breaks down data into approximately 60 risk factors, which are then used to create lists of buildings that are most vulnerable to fire.
Similarly, the NYC Department of Buildings utilizes automation to tackle the challenge of illegal building conversions and hazardous conditions. The Department uses algorithms to sift through datasets, including complaint records, property violations, and permit histories, to pinpoint buildings with a high likelihood of non-compliance. This targeted strategy significantly improved the Department's ability to identify and address dangerous conditions, providing safer living environments for residents.
The use of algorithm-based automation in predictive policing has been a subject of intense debate. Deployed algorithms often use historical crime data to forecast where crimes are likely to occur. However, because the data may reflect biased policing practices, the predictions can lead to over-policing in neighborhoods with residents that are largely people of color, further entrenching systemic inequalities.
In Chicago, the "heat list" algorithm, officially known as the Strategic Subject List (SSL), was designed to identify individuals at high risk of becoming involved in violent crime, either as victims or perpetrators. The algorithm analyzed various factors to generate a risk score for individuals, including arrest records, gang affiliations, and social networks. However, studies revealed that the algorithm disproportionately identified young Black men as high-risk, even when they had no criminal history and had never been victims of a crime. This approach resulted in increased surveillance and policing of these individuals, particularly in neighborhoods already heavily monitored by law enforcement.
To address these built-in biases, cities are reevaluating their use of predictive policing tools, while others are exploring alternative approaches that prioritize community engagement and restorative justice. The city of San Jose took a pioneering step by releasing a set of guidelines to govern the ethical deployment of technologies within the city’s operations. These guidelines ensure automated systems prioritize preventing biased outcomes that could negatively impact historically marginalized groups. Currently, when the city purchases a digital system, it asks vendors questions intended to catch potential bias, as well as what the vendor does to protect against it. But the city needs to develop more tools and protocols for checking bias and equity…” says city’s CIO, Khaled Tawfik.
In the healthcare sector, algorithm-based automation diagnostic tools have also been found to exhibit racial bias. A study published by researchers at the University of Pennsylvania Health System highlighted how algorithms could perpetuate racial disparities in care. Research determined that industry-used algorithms designed to determine which patients would benefit from additional medical care were less likely to recommend such care for Black patients, even when they had health profiles comparable to white patients. This disparity was traced back to the algorithms' reliance on healthcare costs and prior engagement with the healthcare system as a proxy for health needs, which inadvertently penalized patients from lower-income backgrounds.
Data for Black Lives (D4BL) – a movement of activists, organizers, and scientists committed to the mission of using data to create concrete and measurable change in the lives of Black people – is helping address this issue by advocating for using data as a tool for social change, emphasizing the importance of community-driven data practices that center on the needs and voices of marginalized populations. Analysis of automation in health diagnostics reveals a need for algorithms to be developed with direct input from affected communities to guarantee tools used to make critical decisions about health and well-being have an equitable impact.
Ichor serves corporations, government agencies, and non-profits by developing advanced strategies that drive inclusive outcomes. Through deep partnerships with communities and stakeholders, Ichor ensures that data-driven solutions are both cutting-edge and grounded in equity, creating sustainable impact and ensuring communities benefit from technological advancements. Contact us to learn how Ichor can help your organization achieve transformative outcomes through applications of equitable solutions to data-driven strategies.