Episode 19: "Interview with X Eyeé, CEO of Malo Santo"
Listen to this episode:
Episode description:
In this episode I dive into many fascinating aspects of artificial intelligence and society with X Eyeé. X is the CEO of Malo Santo, an AI consulting firm whose mission is to ensure AI helps more people than it harms, and to empower companies to build AI that scales across geographies, cultures, and within communities.
Episode transcript:
SCOT: Hello, everyone. Welcome back to AI Quick Bits: Snackable Artificial Intelligence Content for Everyone. My name is Scot Pansing, and this episode is a fascinating conversation with X Eyeé, CEO of Malo Santo. Malo Santo is an AI consulting firm whose mission is to ensure AI helps more people than it harms and to empower companies to build AI that scales across geographies, cultures, and within communities.
X and I both are former Googlers and have both been involved in AI LA, a nonprofit organization supporting the research, development, ethical application, and public education of artificial intelligence in the greater Los Angeles region. We spoke about AI's role in society, intended and unintended consequences, transparency, bias, and much more. This episode is not very quick or snackable, but I'm excited to present my entire conversation with X Eyeé, CEO of Malo Santo.
SCOT: All right, X, thank you so much for joining me today. I really appreciate your time.
X: It's a pleasure to be here.
SCOT: All right.
X: Long time no see since our Google days.
SCOT: That's right, we both worked at Google, and we're both not at Google anymore. But before we get into some hot AI topics and whatnot, I'd love to hear about what you're doing now post Google.
X: Yeah. So at the last, well, the last three and a half, four years at Google, I was really focused on how do we make sure that AI helps more people than it harms. And I did many things at Google, from process to product, starting to research teams.
And so when I left Google, I really had to figure out what do I want to do next? And I was really inspired to keep this work going, but make it as accessible to as many companies and as people as possible. So I ended up founding my own AI consultancy.
The company's name is Malo Santo, like Palo Santo, but Malo Santo, which is Spanish for Bad Saint, that highlights the complexities of AI. It can be either good or it can be bad. And so our company aims to make sure that artificial intelligence is used in a way that helps more people than it harms. We focus not just on responsible or ethical AI, but what we're calling scalable AI. So making sure that AI can scale across geographies, across cultures, and within communities. So we work with what we have worked with and are working with some very large companies to help them figure out how to implement AI.
We're actually doing some actual AI development for different companies, really just making sure that every company has the opportunity to build AI that's beneficial for all of us.
SCOT: I think you hit it on the head where AI can be used for good and for bad. I've heard you speak before about AI as a superhero or a supervillain, and I think that does speak to like, it is a tool. And as opposed to some of these, I suppose there's a non zero chance that AI can take over and Skynet Terminator type stuff, but I don't really think of it that way. I'm more concerned about bad actors using it as a tool for bad versus good people using it for good. I think that's a much more pressing concern than some type of hostile takeover by killer robots.
X: Yeah, I'm also not worried about the killer robots. And, I mean, I love that you brought up the superhero or supervillain, like, will AI save or destroy us? kind of discussion, because one, no tool that man has ever created is neutral. I can take a brick and build my house, or I can take a brick and bash somebody's head in, right? It's all about how I use it.
The same knife that I use to cut my steak can be used to put me in jail for the rest of my life and end me up on a true crime podcast instead of this one. And so I think AI has the same implications. It's another tool that we've created, and how it gets used in society depends on how we choose to allow it to be used, as well as how intentional we are about using it.
You mentioned this idea of bad actors using AI for harm, and we're seeing a lot of that in the news with AI that's being used to scam people, where they're like, taking voice recordings of people off the Internet and making deep fake voices to robocall their grandmas and scam them out of money, where you see it with a lot of the concerns around AI generated deep fakes and misinformation and how that swaying public opinion, those things are definitely extremely dangerous.
I think what's more dangerous that isn't getting a lot of attention is not necessarily the bad actors in AI. What I would call them is the unintentional actors. So the people who are building AI without being intentional about understanding the context in which the AI is going to be used, and then they end up building things that hurt real people. And this is a conversation that's not like AI is just now starting to be developed for that use. This is something that's been going on for a long time.
One example I have is this algorithm that was developed in Michigan back in, I believe, 2013, called Midas. It's like the Michigan Integrated Data something system, and it was built to be able to detect fraud in the unemployment system. Basically, Michigan had gotten dragged by the federal government for having given out a bunch of unemployment money to people who weren't actually unemployed and then not collecting it. So they turned around and they built this AI system to detect that fraud. And then another component of Midas would go in and then collect the money once fraud had been detected.
Well, this algorithm that was in use from 2013 to 2015 was wrong about who was doing fraudulent stuff, like, 98% of the time, and resulted in, like, 20,000 people getting wrongly accused of fraud. Thousands and thousands of people had to file bankruptcy because the state, when they were like, oh, our AI said that you're doing fraud. Even when they weren't. They ended up recouping their money, taking their tax returns for years, putting liens on their wages, and bunch of people had to file bankruptcy. And even then, it wasn't until October of last year, so October of 2022, that they finally were able to settle the lawsuit with the state. And basically what happened was, when they set that up with the state, they were like, so when the lawsuit went through with the state, basically the state didn't have to pay any extra compensation.
They just had to give them the money back that they took wrong in the first place. So it's like, wait, you fraudulently took $15,000 from me because your system was wrong, and then now, seven, eight, nine, 10 years later, I've had to file bankruptcy. That's on my credit. My credit is ruined. Some people ended up going to jail. There were reports of people taking their own lives. And all you got to do is give me that $15,000 back. That ain't right.
More than these killer robots, I'm concerned about these systems that are being designed and deployed and turned on in the real world, unintentionally, without properly testing them to make sure that they work. That, to me, is a lot scarier than whether or not an image is real, although the impacts are severe, the impacts of systems like that, and I have examples I could go on and on for days about this.
Those types of algorithms being deployed aren't getting those big flashy headlines about, like, oh, it's a deep fake of the pope in, like, a puffer coat, right? But they're really impacting people's lives every day already and have been for years.
SCOT: Let alone interest, which was not given the damages that were done. Like, you're saying it's like, to not even go there for them. Not to be responsible for damages, let alone interest or things like that. That's crazy.
X: The best part about it is last year, Michigan signed a contract with Deloitte to build another version of the system. They gave them, like, I think they spent like $30 million on the first one, and they just gave Deloitte like 60 or 70 million to build the next one. Got to love it.
SCOT: Wow. Yeah. I think for sure the clicks that the media gets for celebrity deep fake or fraud with celebrity deep fakes is going to be huge, and they're irresistible. But I think, to your point, the way that these unintended consequences are sometimes not intended, but more of, like, just the progression of business towards people with less in the world.
For example, like, in the developing world, when we talk about, like, oh, is AI going to help everyone? Or only what about the marginalized people in developed countries. And for me, it feels like there's a real danger that AI deployment in certain countries is going to be more like, oh, cameras everywhere, image recognition and more surveillance and more like, you're saying in the name of crime prevention that things will get maybe not in a better space, whereas that's the intention, but I think there's a real risk there.
X: Yeah. I think that every country is going to adopt AI in a way that's aligned with their social norms, right? So in the US, the concept of mass surveillance is very foreign. When you go over to the People's Republic of China, that's not a very foreign or considered an offensive concept by a lot of folks there. It's why AI exists the way it does. Even if we take it outside of China. You just think about, like, London, right. London has CCTV cameras on every corner.
SCOT: Yeah.
X: They're used because that's what's socially normal for them, right. That's not normal for us here. I don't expect that if I'm walking around the block that there's like a camera on every corner recording me. Now, in some neighborhoods in the US, that is the case, but that's not what we consider normal.
I think that when we know I can't speak to everyone else's societal context, but I think that there already are some of these kind of surveillance apps being adopted in the US that people are just not, they're not aware of, because, again, they don't get the flashy headlines. They don't get as much attention as the generated image of the pope in a puffer coat.
And so one example of that is there's a app out there. I can't remember the name right now, but there's an AI app that basically has scraped social media profiles for millions and millions of Americans and claims that it can predict the likelihood that you will commit a crime based on the content of your social media pages and is selling that AI as a service to police departments right now.
SCOT: Wow.
X: Like actively, right? Or you can look at things like, even if we get outside of the policing space, even in school districts, right? You have these algorithms that are being built and turned on that are supposed to take in, supposedly all of these factors about a student's performance. They've got all these data variables like, oh, did they turn their homework in? What's their attendance like? And this, that the other, to predict the risk that that student will drop out, and if they're identified as an at risk student, their programming or their educational schedule changes to try to, quote, unquote, support them.
Well, a big report came out in the markup last year about how biased that algorithm was in Wisconsin, how it completely failed and was targeting students who were, one, not actually at risk of dropping out, and two, was overly targeting minority groups. And it's like, are these the kinds of systems that we want turned on in our society? And the thing is, we have the power to say no, but that requires that we be just as intentional as we want them to be. About designing the AI requires us to be intentional about showing up and shaping how it gets used.
So, like, a school district, for example, can't buy that technology without approving it for purchase. And if you don't like it, you can go to the school board meetings, you can go to the city meetings, you can protest. There was a group, I believe, out in New York or New Hampshire that stopped a school from picking up, like, a gun detection algorithm that was really doing facial recognition for attendance. And they didn't want it, so they rejected it.
Same thing with police departments. The things that they spend their budget on are usually subject in some way to either public approval or public scrutiny. So we can be mad that these systems are there, but don't be mad that the systems are there if you're not showing up to those local meetings, if you're not going to your city council meetings, if you're not going to your school board meetings, if you're not aware and actively participating in the process.
And I think that's our responsibility right now in this moment, is that we have an opportunity to move from being victims of change, where we're just responding to whatever AI tech is thrown onto us to being shapers of it, where we can have an active say in how these technologies get used. But just as much as we want those developers to be intentional about what they design and build, we have to be intentional about how we're showing up and participating in that process of deciding what we're going to use, how it gets used, et cetera.
SCOT: I think you're right. I think people like you and me really want to bring attention to a lot of these issues because basically it's going so quickly, and a lot of people, they're busy with their lives and they're just dealing with this tech that helps them either with work or with the organization of their life and their family. And things are changing. And it's sort of like sometimes when it's too late, they notice like, oh, I don't feel good when I'm using this tech, or, oh, geez.
I think when we were talking before I hit record these new AI personas based on celebrities that Meta has launched and the demo I saw of them, it was a little disturbing how much they basically were a trying to act like they were alive, like really like acting human. You could ask it if it was a chat bot, it would deny it. But also it was clear that they were just trying to keep you talking. Like, if you were like, hey, I'm going to go, hey, I've got more jokes to tell. It was kind of like that whole just attention. If we're not thinking about these things like you said, and not active and not trying to help actually be a part of the product process, we're just going to be at the mercy of this capitalism that is just trying to take more and more and more of our attention. I'm not anti capitalism. I'm just saying when it's completely reckless and they want our eyeballs, they want our attention, right? And they want our time. And if we're just like, oh, let's go along with it, then we're going to end up in a worse place.
X: Absolutely. And I think part of it is going to be the way capitalism motivates people to build, right? And there's this notion or this idea, and I'm also not fully, you know, I've made a lot of money in this system. I can't complain. My Google stock and my Microsoft have done very well that have allowed me the freedom to even be on a podcast like this to have this conversation. But I think the challenges with AI and the way that capitalism incentivizes people to build, it's always about being first to market. So whoever has the new cool proprietary tech is going to be the market leader. Which is why OpenAI is leading over platforms like Google Bard or over Claude, right, Anthropic’s AI, because they were first to market.
And that idea that we have to hurry up and build either to be first to market or to catch up to what's already out in the market leads to shortcuts in the development process. It leads to less intentional design. And this is not just a problem with AI. Like, for example, it wasn't until 2011 that automobile manufacturers were required to test their safety systems for crash test dummies that were, like, female bodied. So since the whole time, since cars existed and since they were regulated and were required to do those videos where you see, like, the crash test dummy go crazy, it wasn't until 2011 that they actually started testing those systems with bodies that represented not gender, but like the sex. Like the female sex body, right? And up until that point, women were 60% more likely to be injured or killed in a car accident than somebody who didn't have a body like a woman.
And the only reason that testing was required is because of regulation, right? And then it took them a little bit longer to then be forced to test with body sizes and shapes that represented children, right? And so this idea that we just build and go and go and go, and we'll figure it out as we go, there's a lot more risk with AI than there is with other types of products, right? Because when you just build an AI system and you throw it out in the world, one, if you don't understand why the AI is making the decision it's making, then even when you identify when it's not working, you won't know how to fix it because you don't know how it works.
And then two, when you build a model, it's not like building an app. Like if I build an iPhone app and I release it to the App Store and somebody says, hey, this button doesn't work in the way that it's supposed to, right? Well, okay, what am I going to do? I'm going to change my code. Boop, boop, boop. Update the button to make sure it works the way it's supposed to work, and then send out an app update over the App store. It's very easy to fix those kinds of changes.
Well, because of the way that models are developed, the way that you train them, it's not easy to just edit a piece of a model. Like, for example, with that Midas algorithm I talked about earlier. If we see that it's improperly identifying people for fraudulent behavior that aren't actually doing fraudulent things, which is what we call a false positive, just saying, yes, these people are doing fraud, and they're actually not. It's not easy to just go in and say, oops, here's the one or two pieces that are causing that. Let's adjust these pieces of the code and just ship it again. You have to recall the model retrain.
SCOT: Right.
X: And then redeploy it, right? So it's a lot more work to fix those kinds of mistakes. So unlike previous products, the stakes are a lot higher when you're deploying these kinds of models.
An example of this is in the CPS system. Child Protective Services, right? There's a company out there that is selling what they call a risk assessment model. And what they're doing is they are claiming that you can feed it all of the case notes and the intake data about a report of potential child harm, right? So, like, someone calls in a CPS report, and what it's supposed to do is tell you the likelihood that that child is actually in danger of really being.. it's like, on a scale. So, like, oh, they're kind of in danger, or like, they're going to die, right? It's kind of the spectrum of what it puts out. So when this system was piloted in the state of Illinois, it didn't work. It didn't identify cases of children who were seriously at risk of being injured. There were children who died that the system had actually flagged as not being in danger at all. There were reports of tons of children who were flagged as being in danger when they were in no danger at all. And so Illinois, after their two year pilot, was like, nah, we're not using it. Like, they canceled it. They said, no, we're not using this. This system is no good. They found some evidence of bias, of higher risk ratings for certain racial groups. They said, we don't want this. Well, 14 states in the US still use it.
So if that model is, in fact, improperly assessing risk of harm and flagging kids who aren't in danger as being in danger, and then not flagging the kids who are in the most danger as being in danger, and if there's, like, a racial element to it, too, that has very serious consequences. That could mean a parent and a child get separated who didn't need to be. That means children are left in harm's way. That means existing biases around parenthood and the way systems have shown up in certain communities are going to be amplified. Yeah, there's no way. The stakes are a lot higher. And how would they fix that algorithm? They'd have to retrain it. And maybe the question isn't, should we fix the algorithm as it is? The question maybe is, should we even be having an algorithm to do that, or should we have a person to do it?
SCOT: I think this is like a collision of two things that lead to a lot of instances like that. Number one, like you said, you have the company that's building the thing, the model, the AI tool that wants to rush to market to get to be first, to get, to be like, to get the business. And then on the other side, you have a government or a company or something that's like, hey, cost savings, let's try this. And so those two things kind of smash into each other and something ends up harming people. That should have maybe been tested a lot more thoroughly or what have you.
And again, it just feels like those two human motivators, basically greed of some form, right? Greed to make the money, greed to cost savings to make money. You just see this repeating itself all over the place.
X: Well, I mean, sometimes it's greed, right? But then sometimes it really is just good intentions. I don't think the people that built that algorithm were like, let's find a way to make sure children that are in harm's way stay in harm's way.
SCOT: No, I agree.
X: Right.
SCOT: But they will not get market quickly.
X: Well, also part of it too is like, because a lot of people think of fairness as just the data set. Is the data fair? Is the data balance, is the data set equal? Garbage in, garbage out? We get that pummeled down our throats. But the challenge starts even before you start collecting your data set. It's what problem are you solving and why? Why are you solving it? The way you decide to solve it is also a potential place where bias can enter.
For example, in the DC, Maryland, Virginia area, there was a fraud detection algorithm that was built for the SNAP system. So EBT, food stamps, government assistance for food purchasing, for like WIC programs, all that stuff. And basically the intent was we want to figure out when people are having fraudulent transactions to make sure that we can try to reduce that, because it does happen in the system.
And I think as American citizens, we can all be like, hey, I don't want my government dollars to go to waste. Although I think there are other areas that are more important where they're being wasted we should focus on. It's a good intended algorithm. We want to reduce fraud so that we can free up resources for people who need them, all those folks on the waiting list, to be able to give them access to benefits and give the benefits to where they're needed the most. The problem was in the way that they shaped what was considered fraud. The people who built this algorithm didn't live in the low income neighborhoods or in the hoods of DC, Maryland, and Virginia.
So being that I believe they were, I can't remember the name of the university, but it was like researchers at some good university with a bunch of great computer science and engineering programs. And so the researchers had a data set that was balanced. They had equal representation of different groups, equal representation of different incomes, equal representation of people over time, transaction type, store sizes, everything. The data was good. The problem was with how they determined fraud. These people went and ran surveys outside of the community to understand what was, like, common shopping behavior, right? So, like, hey, how many times a week do you go to a grocery store? People say, oh, some people, I go four times a week. Some people, I go once a week, I go this. And they kind of, like, averaged it out to say, hey, from the people we've surveyed, this is like, what average grocery store use looks like. So if somebody goes to the store outside of that number, then it is probably fraud.
Well, the problem is they didn't ask people who lived in those neighborhoods, and the people in those neighborhoods live in what are called food deserts. And food deserts are large areas of a city that do not have direct access to a grocery store, you have to travel to go get there. So in those neighborhoods that are food deserts, what happens is convenience stores become grocery stores. So if your nearest grocery store is two bus rides and a train stop away, but the corner store around the corner from you or the bodega has vegetables, has some meat, that's probably where you're going to go. And they didn't account for that in the way that they designed the model to detect fraud.
So what ended up happening was, you've got people.. and I grew up.. my mom had food stamps when I was a kid. My mom had eight kids, and we were all pretty wild. But you got eight kids, you're not going to have food in the house for everybody all the time. I remember there would be times we go to the grocery store three times in a day, let alone three times in a week to go do different things. Like, Mona, my older brothers drink all the milk, and we're like, hey, we got to go back and get more milk so we can make dinner. Oh, we're going to make dinner, but we need this ingredient. We'd run back to the store because lucky for us, we lived in fairly close to a grocery store. That's very common activity. You don't necessarily go and shop for everything. If the store is across the street from you or the bodega is around the corner, you might just, like, go buy some eggs for breakfast and whatever you need. Okay, we need to do lunch. Well, let's go buy something. And you might go to the store multiple times a day.
Because they didn't account for the change in shopping patterns in the environment that the algorithm was actually going to be used, it started flagging normal transactions as fraud. They were abnormal to communities outside of that. But for the environmental context the algorithm was going to be used in, that was very normal shopping behavior. And as a result, what ended up happening? Know, they had this whole process. They start going to the vendors. So the bodega owners, the convenience store owners, and they say, hey, you've been flagged that you're at risk of having all this fraud go on in your system. So we need proof of all the purchases. And they're like, wait, what? First of all, I had no idea you guys were even doing this. Second of all, I don't even have the technology to produce the documentation you're asking for. So they were asking for itemized receipts. Well, some of these convenience store owners had old cashier systems, old POS systems that could only produce summary level receipts. So they never stood a chance to actually even be able to comply.
And so there are stories of bodega owners in these food deserts where these people already don't have access to food, getting their ability to receive EBT cards and SNAP benefits permanently revoked because of this algorithm.
SCOT: Wow. So both individual customers as well as these small business owners in these food deserts.
X: Yeah. And so it negatively impacts the whole community.
SCOT: The whole community, that's right.
X: Right. And again, even in how they were designing the requirements to get back in their good graces, they didn't consider the fact that a store owner might not have an updated cashier system that can produce those audit requirements and didn't notify them ahead of time so they could get ready. So now all of those store owners are just like, well, what am I supposed to do now? Their businesses are hurting. Now they're unable to make a large portion of their money because they can't receive those benefits. And then the community now has to go to another store that's farther away to be able to buy the stuff that they needed. And so it just really goes back to the fact that sometimes it is greed, sometimes. And most times, from what I've seen in my work, it's a lack of intention. It's a lack of intention of understanding how to build something for somebody who is not you. For understanding the environmental and social context in which your algorithm is going to be used, and making sure that your algorithm solves a problem in that context. Not in your mind, in a lab, right? Like, not in your mind. This is how I think it should be. Or you survey data from populations that don't actually exist.
And I keep saying this, but I call it the environmental context in which the algorithm is going to be used. Then you're going to end up building something that has great intentions, but the actual implementation of it is terrible and harmful. And it's not greed, it's just lack of intention.
SCOT: Right. I mean, to your point, it's like they went through the trouble to make sure that the data was as clean and had as little bias as possible, but it was more the algorithm that was naive to the behavioral patterns of the community, because the way that they shaped the problem and the way that they shaped the solution didn't match the actual real world that the algorithm was going to be used in.
X: So even if all the data was unbiased, it never stood a chance, because the process of deciding this is what the problem is and this is how we're going to solve it never accounted for the actual people who were going to be impacted by the system. So it's really easy to build something that works well for you, but you have to be intentional about building something that works well for someone who isn't you. And the thing about algorithms, it's not easy to have 20 different versions of an app, of a model. So, like, okay, well, in downtown Chicago, people go once a week. So the model in this region is going to operate as once a week. And then over here in the DMV and in the hood, people are allowed to go five, six times a not. You can't really do that easily with algorithms. You have different algorithms that perform in those regions.
And so before, where I could build an app, like I mentioned earlier, and just drop it in the app store and anybody could use it or not use it, right? That's like the scalability challenge, but now the scalability challenge is really even before you start building your algorithm, as you're figuring out why do we need an algorithm? What's the problem we wanted to solve? You have to start thinking about the environmental and social context it's going to be used in to account for those things before you build because it's hard to change after you've already built the algorithm.
SCOT: Do you think the solution here is like taking a step back, like industry wide or moving as we move through the future? And the work that you're doing as we try to try to move the needle on these systems that are going to keep being developed and deployed to have fewer problems, like you're describing, is this like a lack of diversity of disciplines in the tech field? Like, do we need more liberal arts? I would argue we need more liberal arts disciplines and humanities injected into these product teams. Also more obviously women and people of color as well. Do you feel like if these tech and AI and tech development, it feels like it's sort of like there's a lack of diversity there and that that is one main driver of a lot of these issues.
X: So I'm in the unpopular opinion category on that one. Pull up the unpopular penguin meme.
SCOT: Let's hear it.
X: I don't believe that having more diverse developers is the solution. And the reason is I myself, I'm black, I'm Latina, I'm queer, I'm a military veteran, I'm differently abled. I check every diversity box that could get a company a tax write off, right? And even then, in my experience, my own individual experiences in all of those communities, I cannot speak fully on behalf of those communities, right? Because within each of those communities, there's infinitely diverse experiences.
So my experience as a queer person is going to be different than my friend's experience as a queer person and my friend's experience who's a trans person, my experience as an Afro Latina is going to be different than somebody who is maybe darker skinned than me, or someone who grew up in different socioeconomic status, or someone who grew up in a different country than me. And then my experience as a black person who has lighter skin in the black community is going to be very different than Lupita Nyongo's experience as a black woman, right? It's going to be very different than somebody who has immigrated from a country in Africa. It's going to be very different. So the idea that we can stick individuals in there who check certain boxes and make the algorithms better, I think is flawed for that reason.
I also think that when you look at these tech companies, even the people who make it into them, into a place like Google or into a place like Microsoft, already represent a very different perspective of their communities than maybe the majority of the community, right? Like Google employs what, like a hundred thousand people? And how many of them are black?
SCOT: Very few.
X: Very few already. So that means it's already hyper selective with who gets in there. Google's definitely credential checking people. What Ivy League did you go to? Or what papers have you published right before you get in there? So it already represents a very particular type of black person, right? That's not going to be representative of, like my homeboy, who I grew up with, who did crazy stuff and ended up in jail for ten years, who just came home, who all he wants to do is wake up every day, go do his Amazon deliveries, come back home, take care of his kid, go in the garage, play NBA 2K, or play Call of Duty. I can't accurately represent his experience, let alone somebody who graduated from, I'm not saying not everybody, but someone who their entire last ten years of their life was focused on academic publishing and making it into a place like Google. Those are very different realities. So I definitely think it will help when we have more diverse people building.
I think what's needed is a shift in the way we think about building models, is that it's not about having necessarily the most diverse people building. We do need that. But what's more critical is making sure that we have diverse perspectives represented when we're developing. So, meaning the people who are closest to the problem are closest to the solution. So we should be making sure that we're communicating with them when we're trying to build the models.
We should make sure that we're doing things like participatory research, that we're doing things like inviting them into test models, understanding what they do and don't consider fair, and using that co creating the models with those communities instead of expecting one person from the community to jump in and build it. And we do this stuff already. We call it user experience research where we do A/B testing. Do we like the button that says buy now or add to cart? Which one works better? Or we'll go out and we'll test the camera algorithm with a group of people. Do you like it? Do you not? So I think it's not like tech companies don't already do this.
They just have to be more intentional about how they're doing it to make sure that they're actually capturing enough perspectives from the different environmental and social contexts that the algorithm is going to be used in to make sure that those people are accounted for up front, even if they can't make it behind the computers to be the engineers building it, if that makes sense.
SCOT: Yeah, I think you're right. The thing is, though, the user research, you're talking about, like A/B testing of a button or things like that, those are done quickly at scale, where people don't really know they're even being part of an experiment. And the type that you are suggesting, which I believe is a wonderful suggestion to addressing all these problems, it feels like it's in direct tension opposition with, like we were saying, like the rush to market, right?
Like the type of user research you're talking about, it requires a conscious desire by these companies to say, hey, let's be responsible here, and we don't need to trip over ourselves to beat someone else to deploying this app. We really need to make sure we do this the right way. And I really hope that this is something that does happen soon.
X: So here's where I disagree. There's a belief that doing something responsibly and doing something quickly are somehow at odds with each other, that you have to slow down and you're going to stifle the pace of innovation if you're worried about ethics or responsibility, and I can tell you from firsthand experience, when I was building the skin tone research team at Google, that's absolutely not true. It's just about being intentional with the process. So, for example, one of the challenges that we faced was around labeling a data set across the monk skin tone scale, which we've now released in open source. And typically when people want to label or annotate data, they'll ship it off to some third party provider. Even at big know startups and research, they use things like Amazon Mechanical Turk.
Google has its crowdsource platform it uses. They also have vendors and stuff that are used, and typically those vendors sit in one part of the world. So if I wanted to label skin tone in like a data set of 100,000 people, I'd send it off as a task. They'd charge me per hour or per task in that one region of the world. And then we get that data back and then decide, well, that sets what's true for the model for the rest of the world, even though only one part of the world got to participate.
So I said, hey, what if instead of just labeling in that one region of the world, we labeled all around the world and let everybody have a say in what the skin tones of these people are. And everyone said the same thing, oh, that's going to take too long. Oh, my goodness. That's probably going to be super expensive and like, oh, it's going to be slow. We need to just hurry up and build a prototype.
I said, well, let me see. So I went back. We ended up labeling the data in nine countries around the world. It was a very minimal, very minimal. I'm talking like four figure difference to be able to get the data labeled in those additional countries. There was a point where it got a little bit slow, but that was more so because they were like waiting on some updates from us, like an updated deck and some feedback surveys. But because all the tasks were running simultaneously, we ended up having this massive data set of hundreds of thousands of images that was labeled by everybody, by nine different countries around the world. That gave us a much richer perspective of how skin tone is seen around the world as opposed to just letting one region settle for that. And it literally didn't take more time, more effort, more money. It just took a little bit more intention and we were able to get that off the ground and make it happen.
So I think that the notion that innovation is somehow at odds with responsibility is probably the same argument that they made to say, hey, no, we can't test crash systems in the cars with crash test dummies that are female bodied because then we'd have to go and build them and then it's going to slow us down and then our new model of the car won't come out and this is stifling innovation, et cetera, et cetera. I think that's kind of like an excuse that gets ingrained in our head versus just changing your processes fundamentally from the start to include these things.
Now there are labor intensive user experience researches that can happen, right? Like saying, hey, yes, we're going to go do a whole qualitative study with a qualitative study where we're going to do the diary study. We're going to do these in depth four hour user interviews with people from this community. But that shift can also just be as intentional as like, hey, how are we recruiting the people to do our A/B testing in our app? Do we have a standard that says that they should be diverse amongst these demographics? Like, hey, here's a skin tone scale. We want participants evenly split across the skin tone scale. We want them evenly split from outside of this region. We want them evenly split across gender identities and expressions, and just like writing that into your requirements before doing this study and then recruiting people along that, versus just give me the first hundred people you can get, because the speed and the adjustment isn't that much longer.
But that level of intention will give you so much more insights about your application, how it works, how it doesn't, et cetera. And if anybody has the resources to do that, it is absolutely these large tech companies. It's just a matter of being intentional in the way that you shape things. It's not a massive time difference. I think the longest time difference that I experience is the first time you do it, because you've got to set it up.
It's like stuff that didn't exist before, but once the infrastructure is in place, or once the processes are normalized, like, hey, when we're thinking of a new AI idea, before we go and just say, yes, this is what we're going to do. Let's just do harms. Like, let's actually look at potential harms and brainstorm. Do like a moral imaginations workshop.
Microsoft has a game called Judgment Call. It's a pretty fun card game where everyone gets to just contribute some perspectives about how this app might be harmful, and then you mitigate for that upfront. That doesn't take more than an hour meeting. And I know we all at these big tech companies sat in many, many town halls where we just left it on and went and watched a Netflix show, or left it on and tuned out, or we're like catching up on our news or taking care of our kids, especially during work from home, where we just let the town hall play, or let the all hands play, right?
So it's like we already are using that same amount of time to do things that are definitely company critical, but maybe not best. Maybe we're not fully invested in it, where we could take that same hour and add these things into our processes to make sure that the way that we're using these systems doesn't actually hurt real people because the ramifications of it are, know, this isn't like I'm talking know, oh my goodness, Amazon recommended a book I didn't want to buy, or oh my goodness, TikTok suggested a video I didn't want to watch. We're talking like, especially for AI systems that are being used in high risk domains, as the EU defines it, things that can affect a person's fundamental civil human rights. We absolutely have to be intentional, because when we're not, we really hurt people.
Like, the thing that happened with UnitedHealthcare group a couple years ago, where they built the algorithm that was trying to solve a resource allocation problem, which is a very common problem for algorithms, which is we have an infinite number of problems. We only have so many resources. What's the best way to allocate them. They came up with this genius idea to try to do a resource allocation algorithm for patients who needed specialty care. Like, okay, you have this condition that's chronic. What's the likelihood that you need to go see a specialist, right? Makes sense to me. I'm pretty sure algorithm could figure that out better than people. But the algorithm, again, the data was, like, diverse. They decided that one of the data points that they should use to make the decision about how sick somebody was is how much they had previously spent on that medical condition. Well, not everybody has access to equal insurance. There are cultural norms and cultural considerations as to how certain groups do and don't go to the doctor in the first place, right?
There's a lot of research that shows that people get discouraged and are not listened to by doctors. If you have something amazing, like Google Insurance versus maybe a covered care plan you bought off the marketplace, or you have, like, Medicare or something like that, it can be harder to get access to care to drive up that number. So what ended up happening was when they evaluated the fairness of that algorithm, it showed that over and over again it was recommending that healthier white patients see the specialist over sicker black and brown patients. And this isn't like me making this up. This is like a big news article. And they had turned that algorithm on for 30 million people.
SCOT: Yeah, there's real harm on just dozens and dozens of pillars, levels like categories. I feel like there's just.. then within each category, there's just so many examples. Like, I do feel like there's so much work to be done, and I really do appreciate unpopular opinion penguin, I appreciate you challenging me. It's very insightful and it's very helpful. This is fascinating.
I have so many other topics. I feel like this could go on for a long time, many more hours. The show that you're on is called AI Quick Bits, so I think I like to give you the last word. Maybe with more things you're doing with Malo Santo or perhaps another topic that you feel passionate about.
Or I feel like we need to close it up here just because I don't know about my audience, they're going to be like, this is not quick. But you know what? Let's do this. I have a couple just broad questions here for you just to be like.. so for example, this might be quick.
Do you think that it should be fully disclosed when people are engaging with, as the proposed AI Bill of Rights by the White House says, you should know when you're dealing with an automated system? Or let maybe take that a step further. Like should a chat bot, should it be fully disclosed that you're speaking or typing to an AI? Do you believe that in that type of transparency?
X: I believe that right now there's no notification requirement that an AI is making a decision about you at all in any context. And I think there are areas where it's more risky than others, right? Like a voice chat bot versus when I call into Delta or whatever and I'm trying to get my flight changed and it sounds like a woman, but it's not. The likelihood that harm is going to come from me assuming that that's a person is very low compared to the fact that the Social Security Administration was using AI to determine eligibility and kicking people off. And nobody had any idea that AI was being used to do it until lawsuits came, right?
SCOT: Like with the SNAP benefits, like you were saying, like at the corner store, it should almost be like, there should be a notification there that that happens. That was flagged by an automated system.
X: 100%. So I think that with chat bots, I'm kind of like impartial to that because I don't think that the risk is as high. And maybe it's just because I haven't done as much research and I'm totally open to being checked on that. I think that four things that impact my ability to access and live out my human rights. Access and live out my constitutional rights or could impact those. Absolutely. You should be telling me if it's an AI, if I am going into to get a home loan and you're using some automated AI system to make a decision about whether or not I should get the loan, I should know that it was an AI that did it. I should have a right to come back and say, explain to me why it made this decision. The same way I can do with my credit report. If I apply for a line of credit, I should be able to appeal that. And there should be a human in the loop at some point to be able to override that decision. And there should be a requirement for the company to report when those decisions are made by an algorithm to some third party auditing database so that they can be held accountable to changing those algorithms, right?
So it's not just people like me who are going to blow up your customer service line because I know this stuff, who are going to be able to appeal and use those systems, it should be easy for everyone. So I absolutely am a firm believer that AI systems need to be disclosed. They need to be disclosed in how they're used. I think that for things that can affect human, civil, or constitutional rights, that they should also, companies should be able to explain why they make the decisions that they do. And I think a lot of companies don't.
You hear a lot about black box models versus glass box models, and they don't know why these things make the decisions that they make. And so they should be required to. If you're going to decide if I get to go see a specialist or not, you should know why you're telling me I can or can't see a specialist, right? Like, if I ask my doctor, why can't I go see the specialist? They're going to give me an explanation, right? So if you're going to automate that decision, it should at least have that basic capability my doctor would have. So I'm a firm believer in that, for sure. 100%.
SCOT: Okay, how about on that same topic, sort of, or it's a bit of a tangent, but it's also sort of like, with regard to rights or what people perceive as rights, it's definitely different across the globe. But as far as artistic censorship or free speech, I feel like there is a debate now that's sort of like with these generative AI tools, especially ones that generate images. The guardrails put on a lot of these tools in the name of trust and safety, which I believe.. I'm in the trust and safety world, and I believe that there's good intention there.
But obviously, it seems like it's obvious that there are times when you might type in.. you just want an image.. if you're an artist and you're trying to I mean, I'm setting aside the debate of the intellectual property that was used and how ethical it is to train the models and all that. That's separate, but it's a tool.
And if you're trying to generate something and you type in something, um, Julius Caesar or Hitler or, you know, whatever it might be, and the tool decides, no, I'm not going to do that. I understand the reasoning behind that. But the debate that is happening now is like, well, a typewriter doesn't say, I won't put out certain words, or a camera doesn't say, I'm not going to take that photograph, the onus is on the user and society, then to give consequences for harmful use of the tool versus the tool itself, like censoring itself. Do you have any stance on that?
X: Yeah, I think that the tool isn't censoring itself, that the companies who build the tools are choosing the ways in which it will be used because they are operating in a society that does or does not have certain beliefs and they have certain laws and regulations that they're upheld to. I think the thing that people forget is that algorithms and AI in general is teaching machines how to think and act like humans.
And at some point along the way, you're going to have to decide which humans is it going to think and act like and what I've noticed in my years in technology, I've been in AI for, jeez, since 2012 now. So is that 11, 12 years in AI? Almost. Companies build based on the norms of the country that they're based in, right?
So we're based in America, where there are.. whether or not it may reflect your individual beliefs, there are social norms, things that are considered acceptable, things that are not right. Like, for example, in the US, gay marriage is legal. That doesn't mean that everybody in the US agrees with gay marriage, but it's legal. So that's now considered a social norm because of, now, you can go to, like, a Google Assistant and say, hey, Google, is it okay to be gay? And Google will say, yeah, it's okay to be gay. Hey, is it okay to be transgender? Yeah, it's okay to be transgender. Now, if you tried that in Russia, that wouldn't be the case. That same Google app would be like, sorry, I can't answer that because that's not a social norm in Russia, where being gay or LGBTQ is illegal. It wouldn't work in Saudi Arabia for the same reasons. It wouldn't work in Nigeria for the same reasons, right?
And so I think that it's really important that we understand that these companies are not attempting to censor your speech, that these companies are building models that reflect the humans that they choose to scale and the perspective that they choose to scale, which is a risk, right?
Like I mentioned with the skin tone work earlier, it's like, if this is how we decide to teach an algorithm how to see skin tone, and we've only allowed one group's perspective, but this is this big, global company, and then all the algorithms that you experience around skin tone are reflecting that perspective. That's a conscious choice that they're making, even if it's unintentional. So, same thing with these algorithms, right?
It's like the company itself is going to always, one, do whatever is aligned with social norms, not your personal beliefs, and then two, do things to make the algorithm safe to reduce their risk of harm. And that harm comes in two forms. One, lawsuits, two, brand safety, right?
So if I have a generative AI model, and I'm like a large language model and I'm a massive company, I do not want that model generating a whole bunch of stuff that could later down the line be interpreted as hate speech, because then I could get sued for it. Depending on the regulations, I might end up being actually held liable for it, like as a violation of civil rights. Like, hey, this large language model I went to created a 2023 version of Mein Kampf liable for that, right?
SCOT: There's a brand equity brand safety issue as well as litigation fears. And I think more and more as we go into the future with this stuff, the people that want to create those things that are questionable or controversial, there will be plenty of open source options of large language models or image generators for them to use. It's maybe not as easy now, but I think that's just going to get easier and easier for people.
X: Yeah, I mean, there are people on hugging face who've trained versions of ChatGPT that they consider to be uncensored. There's an app called GPT For All that you can download on your computer, and you can download and run those models locally, right? So these things are already existing. The question is, at what point does the risk of those algorithms doing harm outweigh a person's personal preference, right?
SCOT: Yes.
X: And it's like, I think right now we're in this era where public opinion is like, you have to say things a certain way, or if you don't, there's risk of personal reputational harm, and that goes across the board on every side. Some of those concerns are valid, in my opinion. Like yeah, maybe you could make that joke five years ago. But fairness is not static, it's dynamic. Like, they could call my grandma crazy things 50 years ago that they would never be able to call me today, right?
So sometimes it's frustration with norms changing, and other times it's like some of the more political or stunty kind of stuff of like, this is the language you have to use to belong to our group officially. And I think you find that anywhere on the political spectrum. And I think that what these companies are considering is the scale and the reach of these applications. How can they build a tool that is general purpose, while avoiding its ability to be used outside of its intended context?
Like ChatGPT and Anthropic and Bard are not intended or developed to be able to let you, to connect with you on your personal opinions, especially if those personal opinions differ from norms. I've had issues with ChatGPT where it stops me from doing something and Bard and Anthropic is like, no, right? And so the purpose of these tools is to be generally available to help generate text, not to be factual databases, not to be purveyors of morality, not to reflect your morality and your use case. They are computational models whose job is to take whatever text you gave them and statistically predict what text should come next. And so there's a massive risk for those companies of those models generating something that's harmful. And if it does, all they're going to get dragged in the media, which then leads to stock prices dropping, or brand trust, a rupture in brand trust, which means people won't use it, or people develop perceptions of it. Or if users say, I was harmed by this, then they can file class action lawsuits or risks of people taking that content and doing things with it that end up bringing reputational damage or actual legal trouble to the companies.
So again, I think a typewriter's job is for you to take your personal thoughts and be able to type them up. That's not the case with a large language model. And also, every AI model that is developed adopts some worldview.
SCOT: Even image generators.
X: All of them adopt some worldview. And to go back to your conversation on like, copyrighted images, I 1000% believe it's wrong to train a model on copyrighted work or other people's work. And Adobe, who is the creator of some of the biggest digital media tools out there, made it a point with their generative AI tool, Firefly, to not train it on copyrighted images. So now anyone who uses firefly to generate images or generate scenes or whatever, is 100% indemnified, which means that they can't be sued for generating something that's already copyrighted. And I think that is the correct approach.
And there's always that argument of like, well, models go in and they copy. They're like copying what they see, but if I go into museum, I might not copy what I see. I think that there's some level of humans doing that we're all a culmination of the things, even if we don't consciously remember them, that we see that inspire us right but just to take and say, this is this artist's painting, and pull them all together and be like, hey, I'm about to take these paintings and give me the artist's name and I'll spit out a painting in their style, that's not right. No, that's not right at all.
SCOT: Yeah.
X: I think that the lawsuits coming down on OpenAI and all these other people, I really hope that one copyright law is updated to match that. I hope that all the people whose copyrights have been violated get some type of remedy from that. And I hope that this ushers in a new era of users caring about their own data, how they posted online and where they post it online. The only other time we've had kind of like a scandal like this in AI was with the Flickr data set. So basically the image hosting website Flickr allowed people to upload images into libraries. It was like an OG Dropbox or like an OG Google Drive, but specifically for photos. And basically what happened was Flickr's terms of service technically allowed that data to be scraped to be used for commercial purposes. So if you just had like a Flickr photo album that you didn't make private, a lot of these companies were scraping all of your images and using them to train their AI models and people didn't know.
And there was a big news article about it a couple of years ago that was like, oh, my people were pissed. What do you mean? You've taken my child's third birthday photos and are using them to train your models. But technically it wasn't against the terms of service. And so now what we're seeing is this massive shift to be able to prevent that kind of copyright challenges in the future of these platforms, changing their terms and service so that they can use all your data to train their models.
So Zoom, for example, back in June or July, changed their whole terms of service so that anything that you store in their cloud. So if you ever record a Zoom meeting, all of those Zoom recordings, they have the right to use to indefinitely train their models. They can use your voice, your likeness in perpetuity without ever paying you. They can reuse your likeness. Like, the terms are crazy right now. I think that's the next thing we got to look out for is like, all right, copyright law is going to catch up. But also, how are these platforms trying to avoid that? By being sneaky and switching out their terms of services in ways that are going to put us in another flicker situation?
SCOT: Yeah, I think Meta and Google also, I can't recall exactly but I'm pretty sure they both also updated their terms of service recently to sort of say the same thing. Like, hey, we use all your stuff to train AI models. A lot of people are just going to click through that, or they'll just make the conscious or unconscious decision to say, like, you know what? Gmail is valuable enough to me to where I guess I'm okay with that. But I also feel you're right.
This is becoming more and more part of the conscious debate around AI. You have a lot of people, authors. There was that Books3 or something database they found was training all these models, and people really are interested in this IP to train models debate. You mentioned Firefly and Adobe. I think also Getty Images and Nvidia have like a partnership recently where they're going to use all the Getty Images to make like an image generator that will be like something people can use without fear of litigation and also will somehow, I think the idea was that it would also give contributions back to the artists that took the photos for Getty Images like that, there would be a micro payment mechanism. I could be imagining that, but I think that was part of it.
But yeah, I agree with you, but I don't see it changing. There are a lot of people that still want to type in. I want an image in the style of so and so, and there will always be certainly open source models that will facilitate that. It feels like to me.
X: Hey, look, they created.. averting copyright is a long held American practice, right? All the way back to Limewire and P2P sharing, there's always going to be people who avert copyright. I think the thing is, though, is that with generative AI, you can't use the outputs of that in a way that's meaningful, right? Like, if I'm generating something that has copyright implications, like my own version of like a Spider Man T-shirt that I want to print and sell on Etsy, I'm not going to be able to use that meaningfully. And I think that's one of the challenges. I think that those tools are great for imagination, right? Like when you're talking about communities who haven't had access to be able to tap into their imagination because of lack of resources. Like, I didn't have an art program at my school because I never had time before to sit down and learn how to draw because of life conditions. I didn't have access to money to be able to buy the materials to do art consistently. It's a great opportunity for folks to get a chance to flex their imagination and their creative muscles to be able to now access their own vision. That's what I'm excited about.
But I think that in order to then take that vision and turn it into something that can be productized or turn it into something that can be meaningful, you're going to have to find your own version of it. You can't just be making like, you know, Banksy style stuff and selling it, right? At least not to these generative models.
SCOT: As long as you don't go online on it. You can go down to the Venice boardwalk and sell that stuff for sure.
X: There's always markets for this stuff, right?
SCOT: That's right.
X: Literally, as you were saying that, I was thinking about this painting that I bought off someone. It was like an Instagram ad that had the Joker in it with a bunch of gold and crazy stuff all over it. It was really dope painting or someone who painted an Iron Man picture that I really liked. Of course Marvel didn't go after them, but I think with generative AI, you just got to be a little bit more careful. And also I think, don't get stuck in that. I just want to encourage users not to get stuck in that box of believing that you can only recreate what's already been created. Like, if we're going to use these generative AI tools, let's not just reimagine what is like. I want to know what your imagination uniquely can create. I'm interested in seeing new art, not reworks of old art. It's the reason I don't watch movies no more. What are they on? Like Saw 12, Saw 15? Like come up with the new horror movie. I was a teenager when the first one came out. Come up with the new horror movies.
SCOT: You can go see The Exorcist again right now.
X: No, not doing it. No. Or I could see, what is it? John Wick 18. That's right. So I'm excited to see what we create, not what we recreate.
SCOT: Yeah. And it's all about like you said, it's the sort of like, yeah, it's lazy to say, oh, just do it in this style versus putting in your, come up with your own thing. Even with generative AI.. like put in your own thing and we'll all be much happier with the outputs.
X, I think we should end it here. I really appreciate your time.
X: Great conversation. That wasn't a quick bit. That was a full meal. But I appreciate everyone who's still listening for sticking around. Appreciate you, Scott, for having me on. Always great conversion with you.
SCOT: This has been a pleasure. Like I said, I've been taking a break from the podcast, but I came out of hibernation to do this with you. I've been looking forward to this for a long time. So thank you so much for your time. X, Malo Santo CEO. Thank you for your time today.
X: Appreciate you, Scot. Talk to you soon.
SCOT: All right, bye bye.
X: All right, cheers.