Shion_mixdown.mp3: Audio automatically transcribed by Sonix

Shion_mixdown.mp3: this mp3 audio file was automatically transcribed by Sonix with the best speech-to-text algorithms. This transcript may contain errors.

Speaker1:
Welcome to Radical AI, a podcast about technology, power, society and what it means to be human in the age of information. We are your hosts, Dylan and Jess. We're two PhD students with different backgrounds researching AI and technology ethics.

Speaker2:
In this episode, we interview Shayan Guha about how governments adopt algorithms to enforce public policy.

Speaker1:
Cheyenne is an assistant professor in the Faculty of Information at University of Toronto. His research fits into the field of human centered data science, which he helped develop. Cheyenne explores the intersection between A.I. and public policy by researching algorithmic decision making in public services such as criminal justice, child welfare and health care.

Speaker2:
And with that said, we cover a lot of ground in this interview, so let's just dive right in.

Speaker1:
We are on the call today with Shayan Guha. Shayan, welcome to the show.

Speaker3:
Yeah. Thank you very much for having me here.

Speaker1:
Absolutely. And today we're going to be talking about a few different case studies of data driven decision making in public services. And so to start off, I'm wondering if you could just describe to us some examples of data driven decision making and data driven approaches in the public services sector.

Speaker3:
Yeah, absolutely. So this is not a North American thing. This is not a global South. This is not a global North thing. Governments everywhere, historically have always been very interested in using data to make decisions about the people that they serve. Specifically, however, in the past 5 to 10 years, there's been a lot of attention that's been put on how a lot of this data that's been collected at every level of government, be it kind of like a federal or national government or kind of like a state or provincial level or even at a very local city or other subdivision level. Governments have started implementing predictive risk based algorithms to make decisions from all of the data that it collects about people. And so that exists at practically every single government service that we know or care about or don't even know. So, for instance, the ones that get a lot of attention is the criminal justice system, rightfully so, in the popular consciousness in North America. In the past few years, there's been a lot of attention that has been put on how algorithms are used to make sentencing decisions or set bail or provide various kinds of risk based assessments to judges to make decisions, so on and so forth. And a series of journalistic exposés done by ProPublica and other folks back in 2015, 2016, as well as a US Supreme Court series of cases called Wisconsin versus Loomis. They kind of brought all of this to the national consciousness. But, you know, especially in the criminal justice space, that has been kind of punted back to the to the states in the context of the United States.

Speaker3:
So the Supreme Court basically said that this is not the state should first decide whether before we weigh in. And part of the issue, which is worth mentioning, especially the public sector, is the public sector is inextricably tied up with the legal proceedings. So, you know, there's there's no public services that does not have some kind of a legal or regulatory component associated with it. However, the people who manage or run these legal or regulatory services, so your judges, your lawyers, so on and so forth, they don't get any kind of data or empirical training. So think about this as kind of like a weird dichotomy that on one hand the government is interested in making data driven decisions about people. On the other hand, the people who should be making the decisions have no training in what the decisions are. So when algorithms are thrown at them and they're asked to make decisions, then it creates a lot of conflict and friction. And so we see that now in every single government service beyond criminal justice that exists in school, who should go to watch school kind of ideas in California, Pittsburgh, Wisconsin, Massachusetts, Colorado is all about algorithms and child welfare. But then you also have people deciding what kind of services should Medicare and Medicaid recipients need or unemployment services, what kind of money should go to whom, so on and so forth.

Speaker3:
So there's all of that. And again, it's not a us thing. Let's not just be US centric in other parts of the world. For instance, in Europe, in the social democracies of Europe, you're going to find a lot of this kind of enlightened, data driven decision making where it's kind of looked at as a very positive thing that should be done in order to provide the right kind of service to the right kind of people. So the United States, it's often or or even Canada, it's often kind of looked at as kind of like a punitive deficit based thinking, right. That we must have data driven decision making algorithms so as to prevent the wrong kind of people or to. Prevent fraud. Right. That's the rhetoric. Whereas in Europe. Western Northern Europe, the rhetoric is different. We should be using algorithms so that we can provide the right kind of service to the right kind of people to get it at the right time. And because we are good stewards of public money, so we should be doing this right. So I say this to demonstrate that, you know, algorithms in the public sector are not a partisan thing. It's not about a particular political isle. But it's largely viewed as a very positive thing that will only result in better government services. So that's kind of like a broad introduction to where we are at with with algorithms in the public sector down.

Speaker2:
So let's stick with that positive assumption for because I imagine a lot of listeners from various parts of the world are new to this topic and also translating it to their own context. But when we talk about this positive element about implementing these AI systems, what's that all about? What's the argument for using these systems? And perhaps where are those main inflection points for where these systems are being used?

Speaker3:
There are two main arguments that are often given as to the positive outcome of implementing AI systems in the public sector. So the first thing is increasing efficiencies in whichever service is being provided. So every government has a lot of inefficiencies whether or not the government is providing the service properly or not. There always exists a certain degree of inefficiency and you know, generally it's looked at as a positive thing that we should be trying to reduce those inefficiencies so that we are good stewards of public money. That's one argument that's often made. A second argument that is often made. Is that it's going to reduce costs. Now, that is a far more nebulous argument that you're going to hear in a lot of places. You're going to see it written in a lot of public sector white papers. So before governments implement the systems, they write like 10,000 white papers to kind of prove their point so that when they actually involve the systems that kind of point to the white papers and say, we wrote 10,000 white papers about this before we decided to do this. Why are people criticize it and stuff? So there's always the thing about reducing costs, but there's never any justification about how the costs are reduced. But there seems to be a great optimism that we will move to some kind of automated or more automated world where costs will be reduced. But there's never any definitive thing about how the costs are going to be reduced, because even in a fully automated world it doesn't the logic doesn't automatically imply that just because you live in an automated world, the costs are low. It could be very high. Right.

Speaker1:
I feel like Dillon's is leaning over your right shoulder asking What are the positive arguments? And now I'm going to lean over your left shoulder and say.

Speaker2:
Well, what are the negative arguments?

Speaker1:
Because I'm sure that there are plenty.

Speaker3:
Yeah, absolutely. So. I think that. You know, the negative implications. Lots of people already know these things. But to restate the negative implications, there's two broad negative implication. One broad negative implication is that it. Largely reinforces the existing structural inequalities that are present in society. And the reason for that is the algorithms are trained on administrative data that is collected by the state and administrative data that is often collected by the state that is linked to a variety of different databases. Well, I mean, this is pretty well known in the research record will will reflect the existing structural problems. So algorithms in the public sector often have a tendency to create vicious cycles of that type of reinforcement. And it doesn't affect structural inequalities in any way. Like no amount of throwing algorithms will solve structural inequalities. It will it will exacerbate that if we don't kind of look at them properly. That's the second thing, and this is a lot of people have started kind of doing this work and we started doing this work. It's kind of a pushback against that efficiency and cost argument. The idea is that, well, one of the stated objectives for implementing algorithms is increasing efficiencies and reducing costs. But empirical work actually finds that it doesn't reduce the it doesn't increase the efficiency, it increases efficiency, and it doesn't reduce the cost. In fact, it often brings with it. A whole new data driven algorithmic layer to the bureaucracy. So now you've got street level bureaucrats who are trying to make decisions that the earlier used to do based on their own judgment.

Speaker3:
And we should recognize here that human judgment in the government sector is also biased. It's not a it's not a question of like, algorithms are biased, People are not biased. That's not true. People are also very wise. So but it's a given. But people were first making judgments about other people who seek services. And then you add an additional layer of bureaucracy of here. Now, here is all of these algorithmic systems that you have to deal with and you have to kind of make decisions. That's that's another problem. It really increases a bureaucratic kind of process that the bureaucrats themselves don't like. They just have to use it because they're mandated to use it, but they don't like it. No one likes it. They most of the time, you know, the people who are implementing all of this tends to get all of the flak. But I would push back and say that the street level bureaucrats are actually not to blame here. And the people to blame here are actual people at the positions of power. Right. So we have a tendency, for instance, so let's think about the criminal justice system. We have a tendency to say that. The police use ShotSpotter and crime hotspots to reallocate more policing resources to poorer parts of the city, which are more likely to be black or parts of the city.

Speaker3:
And therefore that's a bad thing. And we should 100% hashtag defund the police. But that's not the that's not a very good rhetoric, right? Because the individual police notwithstanding, the individual cop actually does not have the power to choose not to or to listen to ShotSpotter or crime hotspots or what have. That agency is taken out of their hands. They just have to use it from now on. It's another thing that they have to use. And the real blame kind of lies with the people much higher up the chain. Right. So people who are actually in positions of power to move money are. So although the systems in the public sector cost a lot of money. The real power lies with upper level bureaucrats or people, political appointees who are who have power to throw money to this. And often these are pet projects, right? So the some some person higher up thinks that or even in the case of cities, this becomes like a mayoral for instance prerogative. Happened in New York City back in the eighties and nineties with the ill fated broken Windows policy. Which I'm sure many of the listeners are very familiar with. But the idea was that they would go around neighborhoods and collect data about the smallest types of crimes, and then they would prosecute the smallest type of crimes. They would really, really focus on small quality of life crimes such as littering, crossing the road without the zebra, crossing, truancy.

Speaker3:
And this is called broken windows theory. The idea is that if you prosecute the smaller crimes, the larger crimes won't happen for a variety of reasons. It's since been shown that that theory is not accurate. But why was this a thing? Why this specific theory? Weren't there other other approaches that they could have used? Yes. But this was a moral prerogative. In the 1990s, the mayors were very gung ho about this particular policy and they were like, yes, this is a thing. And now I will devote money to it. Right. So again, with everything in the public sector, it's often a question of follow the money. Where where is the money coming from? Where? Who is throwing that money around? Where is the agency? Right. If the state legislature decides that from now on, all child welfare workers are mandated to use algorithms, the blame does not lie with the individual caseworker who is making a decision on the ground. I mean, what what will that person do? You know, that person doesn't have a choice. So that that is what I'm trying to get at, right? That the fact that the negative implications of algorithms, it's not enough to say the negative implications exist. That's a very shallow critique, right? It's a very shallow distance. The yes, it exists all algorithms of negative consequences. Fine. Now, how do you how do you get at that? Right. And so people are trying to admit that. That's a separate question.

Speaker3:
But like, people are trying to get at the decision making processes and the agency of algorithmic decision making because that is important. Of course, it's equally important to kind of look at the algorithms themselves and do things like look at fairness and conduct audits and increase transparency. Those things are one side of things. The second side of things that not a lot of people do is because that's the hard part, right? Because that part ultimately, ultimately, if you go down that road, you have to become an activist. Ultimately, you have to get more people out to vote to change the leadership so that they don't doesn't result in flawed implementation of systems down the line. Right. So I would urge listeners to think about take a step back and think about the meta perspective. Where is where is this all coming from? Why is this all coming from? And kind of avoid the kind of like shallow pointing of fingers, right? Like the common person off the street when they interact with the government, they don't interact with the mayor, they don't interact with the president, so on and so forth. They interact with your your caseworker, your social services person, your average cop on the street. So it's very easy to blame those people. And beyond the individual blames that they have, it's actually more less about that and more about the structural agencies that lead to the kinds of flawed algorithmic decision making that see.

Speaker2:
One stakeholder that we've talked a bit about but we haven't really gotten to depth is industry itself. And I think industry often gets that blame, as you're saying, kind of thrust on them. But industry is a stakeholder in this system. And so I'm wondering in your mind how industry is is playing in? Because my question is, you know, where is this tech coming from in the first place? And if we follow the money, I'm sure there is some sort of relationship between where that tech is coming from and then how it's being implemented at that mayoral or other political level.

Speaker3:
Absolutely. I mean, industry is definitely a stakeholder in the process because so, you know, if anyone here is familiar with government tenders, you know that the tender, the bid always goes to the lowest bidder. So the government is always trying to find who can build a system at the lowest cost. At the lowest levels of government your your city level, county level, other local level kinds of governments. What you're going to see is you're going to see kind of like small, nebulous third party private companies that latch onto these tenders because in the grand scheme of things, while these tenders do come with a lot of money attached, that's not enough for the big guys to jump in. That's not enough for IBM to jump in. That's not enough for Google or Amazon to jump in. Be jumping at like the higher levels Empire State based contracts or federal contracts. So when the FBI wants facial recognition, they go by Amazon's recognition system, right? When. You know, the NSA wants another risk based algorithm to screen terrorists. They go to IBM to like, I'm giving examples like some of these are well done, but others are hypothetical. Exactly. So. So you've got. So that's that word. So if you think about like big tech, they kind of follow that kind of. Right. Like the big national level things that that's happening with all of that. On the other hand, that's not where most of the action is. Most of the action is actually at the local level, state level, that kind of stuff.

Speaker3:
And they're most of the time you'll see small companies like private companies you don't know. No one knows those teams, but they exist. That's where most of the small businesses are. Right. Or now I've seen a worrying trend, at least in child welfare. So it's a child welfare. It's everywhere else. But you've got Silicon Valley tech startups that have suddenly realized, oh, instead of trying to create the next data gap, I could go to local governments because all the tenders are public. I could go search for tenders, I could create some fancy looking system because again, it's not that hard to wow government officials who low level government officials who know very little about algorithms. It's just not. So it's that's where you kind of see the split between kind of like big tech and startups, like they operate at different levels. But really most of the action is happening at the local level. So you got plenty of like, you know, if you do a cursory Google search, I'm sure that you're going to find plenty of Silicon Valley very recent startups who are claiming to change how resource allocation is done for child welfare or unemployment services or what have you. There's plenty of them. And that because they have now realized that that is a great revenue source because clearly we don't have any good ideas for technology anymore. So.

Speaker1:
Let's dive a little bit deeper into that child welfare case, because I know you've done a bit of work on this in the past, so could you just begin by giving a brief description of what you mean by child welfare and then describe some of the data driven decision making that's happened from governments in this scenario?

Speaker3:
Sure. By child welfare, I mean various child welfare systems that exist at state or local levels in different countries that are towards making sure that children. So that's defined by when you're born to 18 years old, 18 years of age, making sure that children are not abused or neglected. Now, I need to point out that 95% of cases in child welfare and neglect and not abuse abuse is actually a very rare event. Just as, for instance, terrorism is a very rare event, but we are all extremely concerned about terrorism. Similarly, in child welfare, everyone is concerned about abuse, but abuse is actually very low. That is just pure numbers. Abuse is very low. What really happens is neglect and neglect unfortunately has a racist, classist kind of sentiment behind it. What is neglect is not feeding your child organic fruits and vegetables, neglect. Some of you might be like, that's a ludicrous thing to say, but that's actually not true because there are many definitions of neglect, and some definitions of neglect are that you must feed your child fresh fruits and vegetables or else. Well, in many parts of North America and other parts of the world, people live in food deserts and you can't find fresh anything, not when you do three minimum wage jobs. So, you know, you can you can start to see where some of these problematic things exist. And they've existed irrespective of algorithms. That's what I'm trying to say, right? These are structural problems that exist irrespective of algorithms.

Speaker3:
So child welfare systems largely deal with managing neglected abuse of children. And in order to do so, they provide a variety of services. One of the services in child welfare is foster care. Child welfare and foster care often like synonymous. They're not synonymous like foster care is one part of child welfare. There are many other. Most of the time children are actually not put in foster homes. They're there. It's called at home placement. You still live with your parents. It's just that a caseworker comes to check up on you. So, again, if you think about the popular narrative and rhetoric in child welfare. It's most of the time it's not the government coming and taking your kids away, but that's what the popular narrative is. Right. Like if you if you really look at the data, that's not what it is. So it's the popular data is at odds with what actually happens. Even if the government comes most of the time you they do at home placements because we don't have enough foster homes in in the country and we don't need that many homes in the country. And obviously they should be taking your child away if you've abused your child. So. So so those things are there. No data has existed in child welfare for a long time. It's not entirely due. They were doing physical risk assessments on surveys for a long time.

Speaker3:
They were making like risk assessment adjudication in child welfare based on manual collected data for really, really long time. The kind of so I mentioned ProPublica and criminal justice system. That's the case that kind of brought it to the narrative. The other one was the one about child welfare into the popular narrative is also a ProPublica Gizmodo Techradar thing. Those were published, but it's really a book by Virgil Eubanks in 2018 called Automated Inequality that brought one of the key studies was child welfare in Allegheny County in Pennsylvania, where they partnered with our fine friends and colleagues at Carnegie Mellon to do some really interesting things. And I should point out that most of the time, I mean, we tend to focus on where things are going wrong. But most of the time, the I would say that Allegheny County largely did the best that they could under the circumstances and actually improved outcomes in a variety of ways. And they've since kind of become more and more successful. But Virginia did point out many, many instances where it went wrong for a variety of reasons, because the algorithm failed to encapsulate something or the other. And part of the problem is that. Once again, going back to my previous point, these algorithms were built using. Bias risk assessment instruments that risk assessment instruments that will ask questions like does the child have access to fresh food and vegetables and all that? And so a caseworker physically comes to your house, opens your fridge and cupboards, and actually sees if what food is there.

Speaker3:
Now, there's no like in those kind of risk assessments you can't like, there's no wiggle room for saying a client lives in a food desert and doesn't have access to staff or client works three jobs and therefore Kraft mac and cheese. It is right. So when algorithms are implemented, a lot of the times because of this reliance on training on administrative data, so data that is easily quantifiable, Do you have fresh food or vegetables in the house? Yes. No, that is a theoretical, non contextual question. Right. So so we the work that we've all critiqued in child welfare and so I've done deep dive in Wisconsin, that kind of comes out of this kind of analysis. So the algorithms that are there, they're based on data that comes from questions like this over a period of time. So that's a theoretical that's not very good. I mean, there needs to be context around the data. And so part of the effort that my colleagues and I have and students and colleagues and I have tried to kind of put a path forward about, okay, well, if you have to I'm not naive enough to think that. If I criticize algorithms, the algorithms are going to go away. That's not how that works.

Speaker3:
So how do we how do we make progress in that work? Right. Let's think about a world where algorithms exist. They're not going to go away. How do we make progress on one hand? We can make progress on the systems of power. That's a that's a different conversation. On the other hand, we can start to think about what could we do with the algorithms. The current algorithms are biased. What kinds of incremental approaches could we do so that we slowly start making things better? That's probably the approach that works best in the public sector, because why would they listen to you? You might be a very good academic, but like why would a low level bureaucrat listen to you and why on earth would they make changes in there like any change that they make is extremely annoying for them for a variety of reasons. Like any change that they make in a process has a lot of consequences down the line because governments are all about like specific processes. So one of the things that we we were saying is that okay, through our this deep ethnography that we did in Wisconsin, we realized that not only are these risk assessments biased, but the caseworkers know they're biased and they intentionally manipulate the data, like what data they put into the risk assessments in order to get different outcomes that actually benefit whatever variety of reasons.

Speaker3:
Then we found out that this actually caseworkers are required to write down detailed ethnographic narratives for each case every time they have an encounter. And that just gets stored in these archives that no one ever looks at. Now being, you know, having expertise in natural language processing, we realized that, well, if yes, no questions or questions that are kind of on a liquid type scale, strongly disagree to strongly agree. If those questions are bad, if those questions are not appropriate for child welfare. Could we get more context from text and histories? Right. And, you know, because that's what a lot of qualitative social scientists and humanities folks do. Get more context from close reading of texts, except we have 25 years worth of like, like ten terabytes of text. So close manual reading is not going to be enough. Are there computational ways that we could potentially kind of get more context around it so that we can stop living in a risk assessment world and we can live in a more kind of strength based contextual world? So that's kind of like the novel direction that people are headed towards. We've worked on that a fair bit. We've shown in a paper published last year a proof of concept about how you can actually start to get at some of these things. Now, obviously there's going to be people who are not going to like write like you could make the argument that why are ethnographic narratives better than risk assessments? It's a fair question to ask.

Speaker3:
I mean, every data every data source has problems, right? So then why would we you know, if if that's the philosophical question that one must raise, then we should also raise the question then why do we believe closed manual reading done by scholars on archival data if that archival data is also biased? So, you know, that kind of gets at that. So again, that's not a useful critique. The idea is that are there useful things that you can get out of that stops reducing children down to a probability or a number, right? And instead nudges caseworkers to to do things that are in line with other positive things that they've done in the past. Right. And it's kind of threading a very, very, very, very thin needle. Right. Because on one hand. You know, you're still kind of using computational probabilistic algorithms, but on the other hand, you're not. Giving the caseworker a probability or a classification score or a risk score, because again, a single metric, a single score like that has problems. A person who doesn't have algorithmic literacy, like a low level bureaucrat. What does it mean when it says that this child is at 53% of getting abused or allegation of neglect within the six months? What does that mean? How do they. How do they make that determination? That determination.

Speaker3:
And again, this is what we've done earlier. It's kind of based upon three different kind of thought processes. One is like, where does this exist in the overall bureaucratic regulations and processes of things? The second is discretion. Human discretion is a huge part of governments, so governments can choose whether or not to take action based on criteria that's called discretion. So the best example of that is you are speeding, the cop pulls you over and decides to let you go with a warning. That is a very good example of human discretion. That is. Interestingly, as an aside, that is actually one of the arguments against having body cameras for cops, because then it would take that discretion choice away from them. Because if anyone can a cop if I were a cop, I would be very afraid that if my body camera was randomly audited in the future and they discovered that I was literally not catching anyone speeding at all, then that that would have implications for my job, Right. I would do it. So again, these things are complicated. Like it's not as simple as like, do this thing and then everything will be fine. So going back to child welfare, human discretion is a huge part of decisions, right? Decisions that happen within bureaucratic processes. When should the caseworker go to the house? How often should they go to the house? How much money is available for mental health needs of the child? How much money is available for parenting classes? The parents must take a lot of the times in child welfare, in neglect.

Speaker3:
I can tell you a curious statistic. A lot of the times neglect happens with and maybe this is a reflection on on the United States, but it happens with like very young mothers, often like teen moms who actually did not want to have the child but were forced, compelled to have the child for a variety of reasons. And now they have the child, but they never had any training. They do not have any support to know how to care for the child. Right. And so it's not as if they're abusing the child. That's far from truth. But like there's a gulf between abuse and and neglect. And oftentimes, you know, like I was 16 when I had when I had a child, now I'm 18 and my I have a two year old toddler and I want to go party with my friends. Right. It's too easy for us to cast aspersions and say that. Why why should you do these kinds of things? Well, when I was 18, I also wanted to party all the time. So you cannot blame someone who is 18 to want to party and leave the. Jelena Yeah, you know, the child can remain know they're sleeping in their, in their cot. Fine. I'm just going to have a quick makeout and then someone finds that out or the child is a toddler so they climb out of the car, they open the apartment door and they're found wandering outside.

Speaker3:
The very, very typical case, by the way. Very typical kids then. Then what? So the caseworkers need to decide. You can just say that, okay, this is now an unfit parent. You can't just make sport decisions like that. So human discretion is an integral part of the process. And finally, the third part of the process is the actual algorithmic literacy itself. So how do you train people to make decisions through algorithms? So if you take all of these three things together, basically what's happening is that reducing a child down to a classification or a number is a bad thing. It's not. It doesn't it's theoretical. It doesn't. It takes discretion away. It's kind of like they don't like it because it causes problems with the bureaucratic processes. And most importantly, they're unable to interpret it. They're unable to interpret it. So, okay, so then why do we want to live in that world? We don't want to live in network. We want to live in. If we still want to live in an algorithmic world, then maybe we should be trying to do things that support the bureaucratic processes, support human discretion, and support the people to have better algorithmic literacy and decision making capabilities. So that's what we're trying to kind of nudge people towards.

Speaker3:
So, you know. Creating algorithms in the public sector is in itself not a bad thing per se. There's lots of places where actually very low tech algorithms are very useful and high tech obviously are also very useful, but. You know, you have to kind of do it with a with a thoughtful way. You have to kind of understand like, what is the what are the needs of the of the system? Like, how do you support the workers who are in the system? How do you support the caseworkers of the system? So again, in the HCI design community, a lot of people have started talking about worker centered design. This is a very specific thing because HCI as a as a field has now really concerned itself with questions of digital labour, with questions of tech labour, with questions of labour that intersects with algorithms, with data driven systems, with information systems. And this is something that we can see that, well, the algorithms were not designed to be worker centred or to be more human centred. And so that's what we want the conversation to go that way, right? We don't want the conversation to be like, and now I will create the most mathematically perfect algorithm ever. Well, that's useless because people use the algorithms. They don't care about your mathematical facts. So, so, so, so these types of things exist, so I'll stop there.

Speaker2:
Yeah. Well, as we move towards closing, I'm thinking about all of this, and it's it seems like a massive system and it seems like it's playing into, you know, social elements that have existed for a while and are still really thorny, that are really hard to pull apart. And so I think for me, because we're going into this human centered area, the question is still how. And I think the question and this is partially coming from a recent legal decision in the US around Roe v Wade, and a lot of folks feeling powerless about how to look at that even from a technology, all that stuff. For people who want to either take a stand or become involved in this conversation around either child welfare or worker centered design. Are there places that people can plug in or are there resources that people can go to to find out more and then again to plug in?

Speaker3:
Absolutely. So first I would like to point listeners to so based on work that we did at other people did in collaboration, the American Civil Liberties Union actually now has a big nationwide campaign about algorithms and child welfare. Their campaign, for obvious reasons, because they are all about defending the constitutional rights, focuses on the negative implications of child welfare, which are which are very important. So people who want to know more, I would recommend go read that ACLU article as a as an interesting kind of standpoint, if I'm allowed to plug my own work, I think that I can put the links in chat. But there's a series of papers that we wrote in the past few years where we kind of looked at the state of the art of how algorithms are used in child welfare in the US. Then we did a deep dive in Wisconsin to find out all of this stuff around bureaucratic processes and where it exists in the decision making capabilities. That's like a more like very theoretical qualitative kind of paper. And based off those two papers, we kind of try to see if there's proof of concept in computational narrative analysis to support worker centered design. So there's a series of three papers that I think that are useful if people kind of want to get on the on the tech side of things, if there are people who want to kind of get it on the on the more kind of social critical side of things, I think the ACLU article is great.

Speaker3:
And then on the more like tech and design side of things, I think this use of papers are very useful. And then along with other co-authors, Cecilia and Michael Mueller and Marina Cogan, we've also written a textbook very recently that just came out earlier this year published by MIT Press, and that's called Human Centered Data Science and Introduction. And that is a very small textbook. It's not like a big fat text, but it's a very small textbook that's meant for people who want to understand, well, how do you do kind of how do you kind of meld human centered design and data science together? It's kind of meant for professional data scientists or designers or like professional students who want to kind of get into that particular work. I will say that it's very encouraging to see at various stages of the government, I keep a close watch on child welfare in both the US and Canada. There are now positions within child welfare that might be very well suitable for people who want a career in government. And we're also very interested in doing kind of like human centered data science things within the government.

Speaker3:
So I feel like the government has realized that just like outsourcing contracts to nebulous tech startups is not very useful and that there needs to be some amount of capacity building within the system. And so increasingly we're finding more job postings like that. Aclu also hires in this space is really an excellent position. I think they filled it recently, but today they are really at the front line of all of these things, like people, people who really want to get it on the activist side of things. Or if you have a legal background and you want to kind of get into the law stuff, I think that the ACLU articles are a good starting point. In addition to the work, my own work that I plug. I would also recommend that people look at the fine series of articles published by many scholars at Carnegie Mellon with Allegheny County. So Alex Jones, YouTuber Maria Arteta have published a series of work and more recently Ken Holstein High Schoo and students Logan, Stapleton and Kawakami have published some excellent stuff around algorithmic decision making in child welfare. That's certainly something that people could check out. So there's like lots of resources that both kind of like the critical social policy activist side as well as tech design kind of site. Check that out.

Speaker1:
Thank you so much for all those resources. And as usual, we will include links to those in the show notes for this episode. But for now, Shayana, unfortunately, we are out of time for this discussion, even though we know it could go all day. So thank you so much for. Coming on our show and also just for the really important work that you're doing in this field.

Speaker3:
Absolutely. Thank you very much for inviting me. Happy to be here.

Speaker2:
We want to thank Cheyenne again for joining us today and for this wonderful conversation. As always, we like to debrief our conversations. And just let's begin with you. What are you thinking?

Speaker1:
Oof, so many things, so many good things came up in this interview. Well, so many juicy things, maybe. I don't know if they're necessarily.

Speaker2:
Good.

Speaker1:
Objectively, but interesting to discuss maybe is a better way to put it. So first thing that's coming to my mind is our our discussion around discretion with algorithms. And this is something that I think is fascinating in the context of going from a human decision making system to an algorithmic decision making system. And I know that, like, there's some people that are super optimistic about emotional robots and and social robots and their capacity to understand the nuance in human language and the complex differences between humans. But I really thought that Cheyenne's example of algorithms in capability of of exercising discretion was a really good one. The one that he was describing about food deserts and how some algorithms that are trying to allocate child welfare resources or trying to determine whether or not a parent is fit enough to be a parent, they take into consideration that the kind of food that you're feeding your child. And what if you don't have access to, like locally grown, non-GMO, organic foods for your children? What if you live in a food desert? The algorithm wouldn't know that and wouldn't be able to exercise discretion and would flag you as a bad parent for something that you don't really have any control over. And that is one of probably like so many examples that are very similar around how an algorithm often thinks in absolutes and has this determination of what is right or wrong or good or bad based off of whatever heuristics the engineers or the designers of the algorithm have coded into the system. And those those absolute heuristics change over time and change between contexts and between individuals, between locations, between so many things. So yeah, that's, that's my first reaction is I think that's a really important point and it's one that makes me feel uncomfortable about the government using these kinds of systems to allocate resources or to, I don't know, decide if somebody can be a parent or not. What about you, Dylan?

Speaker2:
Yeah, I think that's a really good point. I it makes me think of just all the gaps in the social side of this too. So and we've heard this from a number of guests, but when you're deploying algorithms, especially algorithms coming from certain industry sectors that are divorced from what's possibly happening on the ground or the systems that are already in place, in this case child welfare, there can be real harms in that application because in child welfare, we we do have a system that is very complex and. It sounds like it functions, but it doesn't necessarily function for the betterment or already has things figured out, right? Like it's a social system that is imperfect and that has come under fire a lot of times over the course of its history. And so we're we're at this pragmatic point. I think Cheyenne brought this up where it's it's both good and also complicated that these algorithms exist in a similar way, that it's both good and complicated, that these social services exist. Like it's great that people are doing something and there are people in real need, there are kids in real need and really difficult situations where child protective services are really needed. And also there is some harm reduction that we can do here, including in our algorithms, like not all solutions are created equal.

Speaker2:
And so it's a matter of figuring out how these algorithms, which can be powerful tools and also problematic tools, but can be powerful tools for good, where and how we deploy them and how we build them. And then I think a lot of that has to do with that conversation of the people who are designing it. And then you mentioned some of these startups who are trying to both follow the money and also probably do some good in the world and connecting with government and selling their tech to government. A lot of the government agencies who don't really understand exactly what these algorithms are doing or how they work or anything like that. And so then they deploy them and it's puts the question of agency in, it takes it really out of the hands of the practitioners. And so there's all these different gaps in this massive stakeholder environment that then function but really impact people in need. And with all of these, like even talking about it, right, it's overwhelming. And so how do these different threads fit together in a way that really centers the needs and holistic solutions for in this case, you know, kids, but also anyone being impacted by these algorithms being deployed from the government perspective and in public policy settings.

Speaker1:
Totally agreed. Yeah. And I was also thinking about the Silicon Valley example, as you were speaking about who who actually is creating these algorithms, who's creating these systems, who's collecting this data, and what's that expression about capitalism? It's like if you follow the money back to its source, then that, you know, you figure out like what the actual motivation or the goal behind the system was or something along those lines. I totally just butchered whatever that phrase was.

Speaker2:
But I guess the catch you saying.

Speaker1:
The catch, you saying that I just totally made up and probably definitely bastardized. But I think that it's applicable here because if you think about a lot of these government data driven algorithms or systems that are maybe in partnership with Silicon Valley startups or maybe even big tech giants for certain certain applications, if you trace back the the beginning driving motivation for why these systems were created, it's not always going to be to serve society or to make things better for children or to help out the underserved populations of the country. Oftentimes, if you trace it back to its motivation, it's probably going to be to make money, right? And so if that's the underlying motivation for creating these systems, then I think that they're they're almost set up for failure or at the very least set up to incite and create harm for whoever they're impacting. Because if if their motivation is just to make money, then they're not going to care as much about, you know, the maybe the edge cases where the algorithm was wrong and somebody was impacted poorly because, you know, if it's 98% accuracy then sucks for that 2%. But you know, the 98%, that's like that's great news for us. That means we're hitting our KPIs and that means that we can definitely continue to get more funding from the government to keep this project going. But but I'm more curious about the people who build these systems and their motivation from the start is to ensure that that 2% isn't actually happening or that they can qualitatively investigate and research what is going on with that 2% and mitigate that impact and that harm to make sure that these systems aren't unintentionally desperately impacting certain groups of people, which is pretty much always the case. What happens when we deploy these systems in the real world?

Speaker2:
And which is why Cheyenne's impact has been so important while working with other folks who are bringing this human centered data science methods, but also ideas around how to do this well, or about how to be human centered. And the human centered element, at least just I think in our world, it's still people are still figuring out exactly what that means and how to do it. Well, because it sounds really good. But like, how do we actually either bring a human in the loop or like what you're saying, focus on the fringe cases and focus on the folks, the actual society. Element, societal element, while other things such as money might be prioritized in order to even keep a startup afloat. And so it, you know, it might make sense, but still cause harm. It might be reasonable to follow the money and it still causes harm. So I think the question is and continues to be how can we as as practitioners or as listeners, maybe as we think about some of these things, how do we continue to be human centered in our data science but also in our lives? And as we vote, as we engage in politics, as we go out and we try to change the world even in our daily lives, how can we be more human centered in terms of technology, mediation and our technology use? It's just again, it's feels so complex, especially when you start bringing public policy in it. But I think Shane's work gets to the heart of the fact that we need to start thinking about all of these things and try to figure out some sort of solutions because there's no going back to before algorithms got involved in public policy. They're they're here, they're there. And so now what do we do with it?

Speaker1:
Definitely. And to to quote Cheyenne himself, how can we stop reducing kids or humans down to a probability or a number? How can we scale these systems without reducing people to something that is, it doesn't capture the complex nuances of what it means to be human. So as always, we could keep talking about this for so long, but we are at time. So for more information on today's show, please visit the episode page at Radical Air dot org.

Speaker2:
And if you enjoyed this episode, we invite you to subscribe, rate and review the show on iTunes or your favorite pod culture. Please please do review the show if you're listening right now. Just just go. Just leave us a little a little review. It really does help us get into the. Years and downloads of other folks who might be interested in the topics that we cover on the show. Please catch our regularly scheduled episodes the last Wednesday of every month with possibly some bonus episodes in between. Join our conversation on Twitter at Radical, a iPod. And as always.

Speaker1:
Stay at radical.

Sonix is the world’s most advanced automated transcription, translation, and subtitling platform. Fast, accurate, and affordable.

Automatically convert your mp3 files to text (txt file), Microsoft Word (docx file), and SubRip Subtitle (srt file) in minutes.

Sonix has many features that you'd love including enterprise-grade admin tools, powerful integrations and APIs, upload many different filetypes, automatic transcription software, and easily transcribe your Zoom meetings. Try Sonix for free today.