Episode 14: "Interview with Andrew Gamino-Cheong, CTO of Trustible"
Listen to this episode:
Episode description:
In this episode I speak with Andrew Gamino-Cheong, Co-founder and Chief Technology Officer of Trustible, an artificial intelligence governance management platform. Trustible recently published a whitepaper titled “Generative AI Risks & Considerations,” and the bulk of this episode reviews this content blow-by-blow.
Ignore The AI Utopists And Doomers: The Need For Louder AI Pragmatists
Whitepaper: Generative AI Risks & Considerations
Greg Rutkowski Was Removed From Stable Diffusion, But AI Artists Brought Him Back
What Socrates Can Teach Us About AI
Samsung Bans ChatGPT Among Employees After Sensitive Code Leak
Inject My PDF: Prompt Injection for your Resume
(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
Blueprint for an AI Bill of Rights
Director's Blog: the latest from USPTO leadership
Episode transcript:
SCOT: Hello everyone, and welcome back to AI Quick Bits: Snackable Artificial Intelligence Content for Everyone. My name is Scot Pansing, and today I'm going to be speaking with Andrew Gamino-Cheong, Co-founder and Chief Technology Officer of Trustible, an artificial intelligence governance management platform.
Before we start, I want to say that this podcast audience is really growing to the point that I'm starting to field a pretty consistent stream of requests from technology business leaders to appear on the show. It's very humbling, and I have some wonderful guests lined up. But Andrew, I'm pretty sure I approached you. Welcome to the show!
ANDREW: Thank you. It's a pleasure to be here.
SCOT: Before we get into things here, if you would please give my audience a quick rundown of Trustible and the sorts of services you provide.
ANDREW: Yeah, so Trustible at its core, we're on a mission to help organizations adopt trustworthy and responsible AI. Our name itself is actually a portmanteau of those two terms. Really, the whole idea is that my background has actually always been looking at the intersection of AI and policy, going as far back as what I majored in, in undergrad then some work I did before basically applying AI into the policy space.
And a big thing that we were seeing was a lot of focus on, obviously, laws encouraging organizations to adopt responsible and trustworthy practices, making sure that risks and biases in those systems are identified mitigated as much as possible, and that there's just some general ethical principles that organizations are applying to their use of things.
In particular, we are looking at a broad survey of upcoming regulations like the EU's AI Act Best Practices frameworks, like some stuff being developed by ISO and the NIST AI Risk Management framework, and really thinking about how do we turn those kinds of guidelines, regulations, standards into very actionable steps for teams to do. Some of the insights that we had is like, even if an organization is only using, quote, low risk AI, there's still a process you need to go through to establish that, because it can have legal implications around that.
So that's broadly the mission that we're on. Both my co founder and myself have a background in kind of this policy landscape, and I myself have a lot of expertise in AI, particularly around natural language processing. And so we think that it's been a really great journey so far in kind of helping organizations navigate and adopt those practices and prepare to be compliant with some of those regulations.
SCOT: Yeah, this year is certainly getting busy and setting the stage for it to get crazy busy in the AI regulation and policy space, for sure.
ANDREW: Yeah, we've been seeing and tracking a lot of the different stuff happening, not just at the federal and international level, but also at the state level, whether it's sometimes laws coming out of the New York City Council around use of AI and hiring algorithms, or actually even a regulation in Colorado governing the use of AI for life insurance capabilities. There's a lot happening.
What's interesting is a lot of the differences between some of these laws in terms of what they're looking for and some of the ethical principles being forced into them. And we are seeing a very real possibility of mutually exclusive regulations. And so while the federal government has been slow so far to take definitive action, they might have to come up with some sort of preemptory regulation if they want to prevent a lot of conflicting stuff happening at the state and local level.
SCOT: Yeah, I can definitely see that as a concern with this fragmentation at the state and local level. I think there's also a lot of facial recognition legislation happening at the state level, like Illinois, and some privacy stuff, obviously, in California, that's probably for another episode.
But one thing I wanted to get into before we get into some of the points of your white paper, you have a blog post on the Trustible blog called Ignore the AI Utopists and Doomers: The Need for Louder AI Pragmatists. And it really resonated with me. I've been discussing AI safety, policy and ethics more and more on this show, and I definitely feel like… look, maybe Skynet is a possibility. I guess it's not a 0% chance, but that doesn't seem like something that I personally am going to put any effort into trying to prevent or really think about all that much, because there's so many short term AI concerns that are just happening right now that we can get into.
So I thought that was a very interesting piece. You want to talk about how that blog post came to be? I think it was also published in Forbes, am I right?
ANDREW: Yeah, that's correct. The broad idea behind that is that there's a lot of talking heads out there who really want to just push the most extreme narratives because it gets great press. And so there's a lot of people out there who are focused on existential risks related to AI. To your point, yeah, there is some of that risk. Perhaps there are certain use cases of AI in robotics where interaction with a physical world could have serious effects that should be looked at. But there's also a lot of real, smaller, almost humbler, but very serious risks that also exist on the other side. There's definitely some people who use ChatGPT for the first time and they're so amazed and they're like, the entire world is now different.
People haven't realized it yet, but they haven't actually understood some of its serious limitations, some of the areas where you can easily fool it or bypass some of its kind of protections around it. And so it's still a very early technology. And so saying that everything in the next year or two years is going to be taken over and we'll enter into this post work state of things where AI will take over all tedious jobs is clearly overhyped.
There's still actually a decent way to go technologically as well as culturally and socially before we get there. Both sides can take up a lot of the just media space and thoughts around stuff. And in the meantime, we're actually ignoring how even more basic harms can happen, what the risks are today. Sometimes risks and harms are actually happening because people didn't ask the right questions. The team developing it didn't say, “Hey, is our data representative of the population that we're trying to target?” Then the person buying that system didn't ask that question when applying it to a new context.
There's a lot of people who they actually understand what some of the day to day risks are, and we need them to be more proactive. I think there are a lot of potentially really interesting, positive things about AI. And I think like the existential doomers ignore some of the benefits and the utopias are ignoring most of the risks. And there's kind of that happy medium, which is a more pragmatic approach. AI is going to be inevitable in most fields. That's just a trait of change in society and technology. And yet we do have the ability to say, all right, I will only adopt AI if and when you can prove to me that the benefits can outweigh the risks.
And here's what that looks like. The more we can do to turn our conditions for what is beneficial and acceptable AI, even converting them beyond high level statements like do no harm into like, here's exact tests and criteria to pass honestly, that creates a more actionable pathway that helps us gain the benefits of AI while identifying and mitigating the risks. So we see ourselves as kind of trying to help with that is, say, what is the condition? We say, “We will adopt AI for this if and when it can do this and pass these tests.” And that creates actually a more collaborative, constructive environment that helps us actually benefit from AI.
SCOT: Yeah, and let's get into some of the specifics. But also I think I would comment too.. I don't know whether it's just my newsfeed or whatnot, but I feel like I see a lot more of the doom and dystopia than the utopia as far as the clickbait.
ANDREW: Check out some VC blogs.
SCOT: Oh, yeah, okay. That's probably where I should go to get more positive spin on it.
Well, you have also this whitepaper, Generative AI Risks and Considerations, that I think you put out pretty recently, maybe in June. And I would love to really dig into this. This is going to be basically the bulk of this episode because I think it's got just a ton of really valuable and insightful content in here. First, do you want to talk about how this white paper came to be?
ANDREW: Yeah, so obviously one of the most common questions we get is really how do we handle the risks of generative AI? Plenty of the early groups that we're working with, they initially banned all generative AI because they're working in kind of heavily regulated sectors. Finance, education, healthcare. And so initially, they needed to make sure that they were adopting things in the right way. But then they're trying to say, everyone's telling us we have to adopt generative AI. We can see so many benefits. So again, to that point, how do we adopt this responsibly in following what a lot of groups have been discussing?
If you're going to apply risk management to generative AI, you simply need to know what are the risks inherent to that. And there's already been good understanding of cybersecurity risks for things. Even some of your when you've trained your own data models, you've had access to the training data sets, and a lot of risks come from there. In this world of generative AI, only the biggest and largest companies have the money and the funds to build them. And so now there's actually a heavy element of what is unique to those things that organizations should be thinking about and trying to work to avoid. Some of them are very technical things. Some of them are very even down to the user experience around it or other kinds of system level things going into it. The biggest fear that we have is that a lot of the harms that can come from generative AI will come from ignorance, not necessarily from malintent, where it comes from.
Somebody assumes that, for example, Chat GPT was being updated with every news article published out there, when in fact it knows some stuff. But most of its core training data is actually a lot older. So knowing that and how that risk, for example, can impact things is really important. So that's why we want to actually come up with a list of what are some of the key things that we've thought about and encountered, what are some possible ways that you might test for them or try and mitigate them so you can have a default set of guardrails around this?
And we're really trying to focus on some of the non obvious stuff. Let's go beyond hallucination. Let's look at things. It's not necessarily all of the legal things. We want to be clear that it's not like legal advice, but it does touch on some of the sensitive legal issues that are well discussed.
SCOT: Yeah, and I like how you have it in three buckets. Here we have data risks, model risks, and user experience and context risks. I'd love to really just drill down into each one. Under data risk, the very first one listed is definitely a lightning rod.
Out here in Los Angeles where I'm based, given the Hollywood strikes and kind of in general.. not all artists, but there's a pretty big, I would say, backlash in the artist community against some generative AI, especially writing and images. And the first risk source you have listed here is copyright and IP violations.
And this one is the angle is more about not on what comes out, what is generated, but what is used in the training model. And perhaps people's work was used without them being compensated and put into the model. Do you want to speak about that a little bit and maybe like the legal status of kind of fair use and how that goes?
ANDREW: Yeah, so obviously you're right that this is going to be one of the biggest issues out there, especially early on as everyone's looking at what data is going into these. So yeah, it's basically no secret that there is a lot of copyrighted information that has gone into every large generative AI model out there. The companies themselves aren't really denying this or they'll give noncommon answers on that. And admittedly, on the other side of it, it is also very difficult to determine and detect copyright information, mainly because search engines have been allowed to use it.
And so you could just say, like, hey, I did a Google search.. I didn't bypass any firewalls, I just collected stuff that I was able to find on Google. And then they're trying to say, oh, Google's responsibility is to tell me if it's copyrighted or to have some other indication. And these companies simply haven't put the effort into that. And that is because they're using the same argument that Google does, which is that collecting and using this data constitutes fair use.
Now, I think Google just providing links to things is something generative AI and use of this data in training feels very much different. It feels like a bigger danger. And in particular, actually, the people who had their data on Google were able to benefit from this. And that's I think the biggest shift. If you had a book out there and a snippet of that was on Google and then that led people to buy your book, you as the author were able to benefit and so you're happy to actually have that happen here. There isn't that kind of sinking back to your own work and now it's very exploitative. I think that's the biggest difference that people are really looking at and thinking about.
Obviously, I think there's going to be differences in the legal opinion on use of copyrighted data for model training across different jurisdictions. We can easily see this being one of the biggest points of contention between the US. And Europe as their laws evolve, where the US, in order to encourage AI, might want to still permit certain uses of this data under its fair use doctrine. But the fair use doctrine, to my understanding, isn't even a thing in European law and so it'll look very different there.
SCOT: And Japan is even different, I think, too. And that is definitely an interesting concern. And also, like you said, it feels different. It feels a little more exploitative, but also the different types of generative AI text versus image or video, generative AI, it can feel a lot different.
And I'm not a lawyer, but asking Chat GPT to summarize a piece of literature and getting something back, I can see that that's, like, one concern, and I don't know where that would go. But it feels to me like the optics of the generative image stuff is a little different. Whereas, like, this Stable Diffusion lawsuit with, you know, images are coming back that are generated that have, like, a garbled Getty Images watermark and whatnot. It feels different than, like, a summary of a piece of literature in text.
So I think that a lot of legal work and it's just only begun, I think, with the copyright and IP violations.
ANDREW: And that's why one thing we recommend is if you want to be really conservative.. it's really cool that you can tell Chat GPT to write a poem the style of JRR Tolkien, but as soon as you do that, you're actually creating very much a very clear paper trail of, like, hey, I'm deliberately trying to knock somebody off and not really give them any sort of cut of anything I generate from this.
So we're like, if you literally use somebody's name that is being preserved somehow, somewhere, and that could be used as evidence of intent to try and knock off somebody's work or something like that. So that's like a really pretty extreme but also very conservative guardrail you could put in place.
SCOT: Yeah, it's interesting you mentioned that because I know some people that use Midjourney and they make it a deliberate point not to do things like that. They have made their own sort of manifesto on how they're going to use the tool responsibly. But also, to this point, I saw an article the other day about how in the open source world, like Stable Diffusion's open source, there is an artist who was like, hey, I don't want people to be putting my name into Stable Diffusion and having work come out.
And the creators of Stable Diffusion complied. They kind of were like, okay, we're going to go into the data set and help you out. But because it's open source, the community kind of just put it back in against his wishes. They're like, no, we're going to use the name. I think it's Greg Ratowski. Anyway, I'll put the link in the episode notes, but we're going to keep moving on here. It is called “AI Quick Bits,” but I still feel like this is going to be one of my lengthier episodes. And that's okay because this is really good stuff.
The next data risk in the whitepaper is training data timeliness, which you mentioned that some people may think, hey, this is just fully up to speed on today's news. And I can ask ChatGPT or any of these large language models or even an image generator, like, to generate an image based on something that came out last week. And I think that's good to know that that is not always the case. And I know we'll talk about hallucinations later, but I think that kind of kind of plays into hallucinations or just like maybe things that aren't necessarily entirely true coming back. You want to speak about that a little bit?
ANDREW: Yeah, so the training data aspect really kind of boils down to if there has been a lot of shifts and changes in the world since kind of the bulk of the data was collected. A model that doesn't know about that could be giving you very different kinds of advice or insights and you may not know or understand that.
The best example would be you can easily imagine like an entire model trained on every piece of medical literature that was ever published right up to December of 2019. And in a sense you'd have a huge data set. It'd have great insights into Alzheimer's and diabetes and the flu and it would basically throw an error if you asked about COVID-19 and the whole idea of any sort of chat bot not knowing about that in today's world seems pretty crazy really. This is where the use case and context becomes sensitive.
If you are trying to say like, hey, I'm going to use this to evaluate stocks, for example. Stocks, you would want really up to date information to understand micro macroeconomic trends and these systems simply wouldn't have that. So just simply knowing how the data is being updated is a really important part of that.
SCOT: Yeah, like you were saying, education is a big part of this for people that are just using these tools day in, day out and everyone will start to use these tools more and more. These are definitely things to keep in mind.
I'm going to keep it rolling here with domain knowledge. This is sort of related to the timeliness, but basically I guess a lot of these models are not necessarily fine tuned, right. And people are sort of like, oh, I can just talk to ChatGPT about anything, and talk a little bit about how that may not be necessarily the case.
ANDREW: Yeah, I always assume that ChatGPT is like a glorified chat bot on top of Wikipedia because that is one of its core data sets used for it. And so just like Wikipedia, it's got sometimes a good amount of information about a lot of subjects, but it's never like the best in depth source, particularly if you want to use it for a specific domain.
So the good example there again might be the medical space. It can tell you a little bit about COVID-19, probably about as much as was on Wikipedia or news articles. But if you're trying to understand which kinds of protein spikes have been investigated, the stuff that's really protected in medical journals and that kind of deep research, it doesn't know about that.
Now the question is whether you know that it didn't know about that or not. And that then goes into what assumptions are you making about how in depth this might be versus other stuff?
SCOT: I sort of feel like we would have seen more really accurate, finely tuned models by now. And I'm curious to know your thoughts. I sort of feel like one thesis I have is that the cost in computing power and time and money is just quite high. Like, if you were to say, okay, I'm going to build an LLM that does know everything up to date about modern medicine or all human music, like a musicology professor all the way through everything with modern all forms of modern music.. I guess maybe it's just really costly to fine tune models in that way.
ANDREW: I think it's costly. I also think the organizations that have the proprietary access to the kind of specialized corpora oftentimes tend to be larger and older companies. But Bloomberg, for example, already does have a fine tuned model of GPT trained on basically all Bloomberg's data. I think Thomston Reuters is now working on something analogous for the legal space. I'm not as in tune about some of the stuff in the medical space, but I wouldn't be surprised if their work groups already working on it. I think a lot of them haven't been as public about that, but they definitely have already been looking to incorporate it in the short term into their own systems and are beta testing it in their own platforms before they release it more broadly. And I think that's actually probably the right move because you don't want to give an expert AI system to non experts. That adds in a risk that we'll talk about later.
SCOT: Absolutely. I think you're right. A lot of that stuff is coming. And I think yeah, giving a broad use to the entire company, to some of these expert data, I think also is something that yeah, people won't necessarily want to do. If you're a Fortune 500 company, you don't necessarily want to give everyone in the company access to all of the data everywhere. It's probably a small team that would need access to something like that.
Let's move on; we have two more data risks here, societal and historical biases. That's a definite hot topic right now. You want to get into that?
ANDREW: Yeah. So this, again goes back to sometimes if you want a lot of data, you have to go pretty far back in time and then you're starting to pick up on a lot of stuff that used to exist that ideally should exist less now, but still very much exists. So, for example, if you ask ChatGPT to tell you a story about a hypothetical CEO, it'll likely choose a man's name and use him pronouns for everything. That's because if it went through its data set and said, oh, I go back as far as maybe the 1980s and content and 95% of CEOs always use the he pronoun. It's probably even a higher number within its general corpus. And because it's a probabilistic machine, 95 times it'll throw out the word he when talking about a CEO. And then that is perpetuating a bias that only men are qualified or should be, or might be, or ideally are those kinds of executive positions.
There's a lot of probably even better examples of that deeper down. And honestly, the research on how those biases can show up in images and in text is actually, unfortunately, relatively new. Some of the first papers in this stuff only came out within less than ten years ago, I think. And so there's a lot of other kinds of ways that I think can be very worrisome embedded in that.
SCOT: I think in addition to this huge issue of all these historical biases manifesting themselves in these models, I think another interesting part of bias is like the computational bias that can happen where, for example, I think I read something where a model was fed many images from doctors of skin growth and lesions. And it was like, let's see if we can get it to determine which ones were cancerous. And the model came back with like, oh, if there's a pencil pointing to the lesion, then it's cancerous, because that would be the way that eventually the photo would be. Like a doctor had already determined that this one was a cancerous lesion, so they put the pencil on it, but they wouldn't put the pencil if it was just eczema or a mosquito bite or whatever. So the computer is just kind of like, oh, I got it. You put a pencil, lesions with a pencil pointing to them are cancerous. So I think you have those bias, is just many layers of bias that we're dealing with here.
Finally here with data, for better or worse, the world seems to be moving towards English as the global language. And it sort of feels like, I think even I can't remember who it was the other day, but basically the new computer language, computer coding language is English, and everyone's invited or something like that. I think it might have been at SIGGRAPH from Nvidia. But maybe you have some thoughts a little bit about the pros and cons about language models being trained predominantly on the English language.
ANDREW: Yeah, so I think a couple of the companies have started doing a better job at simply disclosing how many of our documents are in English versus French and Spanish and Chinese. There already had been pretty intense biases before this in natural language processing towards English as well as other than European languages that share similar grammatical and syntactic structures. Plenty of systems I had worked with before in libraries, they would support European languages and English, but then wouldn't support Asian languages like Chinese, Japanese, Korean. That's because there are actual differences in what every character represents. For example, how you actually structure and separate out words and concepts, different conjugation of things. There's a lot of interesting stuff with biases in some languages, with feminine masculine conjugations about stuff there as well.
But the broad part is that most of these systems are primarily trained and work best in English. Some of them have some impressive translation capabilities and can generate stuff in non English languages. But there hasn't been extensive testing on how well they handle inputs and prompts, not in English. And I suspect actually the quality of that is relatively poor. And so then, yeah, it is reinforcing that even if you're not actually fully proficient with English, that itself presents some risks. Because if you're not then telling it to do something in a way that a native English speaker does, it itself will be tripping up on how it's interpreting stuff, and that can cause itself confusion and potentially harms.
SCOT: Yeah, and going back to bias, if we're dealing in a predominant English speaking ecosystem, that certainly would introduce certain potential Western quote biases or it's not guaranteed, but it feels like that is something that could also manifest itself as a result of this situation.
Model Risks is the next big category, and you lead off with one you mentioned a little bit earlier, and it's definitely a big word that everyone throws around with AI, which is hallucination, which I actually have a little bit of an issue with because something that I feel is really interesting with generative AI is how a lot of people are tending to anthropomorphize these systems and sort of.. I mean, I get it, ChatGPT can feel and other large language models can feel like you're interacting with something that's alive, but we're not. And I think that just the word hallucination, as opposed to something like, I don't know, false truth or errors or something like that. It already gives a little bit of an air of like, this thing is living, but nonetheless, we'll set that aside. You want to talk about hallucinations in generative AI systems?
ANDREW: Yeah. So the entire second category here of model risks are really things that tie back to how the model is built, how it's constructed and trained, and some of the things inherent to this generation of stuff. Some of the risks I talked about before around data can be very much applied into other areas as well. Hallucination, and especially hallucination in large language models with generative text. AI is definitely one of its biggest challenges and that is simply where the model is.. it's just looking up probabilities of words right? Given this previous sentence, given the context of what you told me before, I'm going to just predict the next word, or actually it's less than a word. It's even the next character to write sometimes. Now, the problem is that it doesn't have a concept of the world, and so it can easily make up things that simply aren't true.
We've seen a couple of now kind of fun headline examples of this, right? Court cases that cited cases that didn't exist. There is a lawsuit going on because it said that somebody committed a crime that they didn't actually commit. Or actually, the most common thing is that two people with the same name, an answer could combine information about both of them. It's picked up from the web because it's not able to distinguish the two entities between each other. That's like a very common kind that we've seen.
I think it is definitely one of the biggest challenges because it can generate something that seems so right and sometimes it even uses the right words, but the right words in the wrong order changes the meaning. And it gets pretty clear pretty early on that it's not really able to understand the meaning of things a lot of time, so much as like, these words should go together with some nice constructs in between them to make it fluid language. And so it is definitely one of the number one risks of using it. We recommend only use it in areas where hallucinations won't cause kind of serious harms to somebody.
But also there's different ways you can use now to test that and to reduce that. Whether it's throwing a vector database on top of it, setting some different settings around it, obviously having humans review any of its outputs, those are some of the right ways to start monitoring it. We ourselves have done a test where we had it generate dozens of iterations of the same thing and then looked very meticulously to try and estimate how often it was hallucinating for a use case that we had, we're writing up some kind of guidance on what a manual process like that might look like.
SCOT: Yeah, I've had more than a handful of people, probably even more than a dozen, tell me that they have had ChatGPT create false sources like citations that are just completely made up. I think it's getting better, but that definitely is like something that it's not just one or two people have told me that, but I think that has been a relatively common thing.
Another thing I'd like to mention here, since you said these are sequential models, it's predicting a pattern. And I think not knowing is an interesting thing. There's an article in TIME recently by Carissa Véliz, What Socrates Can Teach Us About AI. And actually one of my most recent podcast episodes is called “The Value of Saying I Don't Know,” because Socrates was like, hey, it's good if I can admit that I don't know something. I mean, I'm doing a terrible job of paraphrasing here, but basically asking questions to learn about what you don't know and the Socratic method and that's where wisdom is truly derived, right? And so these sequential models, they have no idea what they don't know. They're not aware of not knowing something.
So, yeah, if it's just going to predict the next word or character. And I think there was an article recently about how, I can't remember which one it was, but someone was saying, like, this is going to be a really hard problem to solve, maybe even unsolvable with the way that since they are sequential models, this is a lot harder problem than people think.
Okay, moving on to input data privacy. I know some companies are kind of like, hey, we've made it so that you can toggle a button and we won't necessarily use all this data, but talk a little bit about the things that you to be careful about what you might put input into generative AI systems.
ANDREW: Yeah, so this is the most common thing we hear about from a lot of large enterprises looking at this. They have data that they are required by law to protect in certain ways and simply passing that data into another platform that might record it, then kind of expands the footprint of that and could actually have some pretty serious reputational and legal implications.
This is actually the area that got ChatGPT banned in Italy initially because they were collecting a ton of information and logging every inputs to the system. And if that was sensitive information like, hey, ChatGPT, I've got these symptoms for something, am I sick? That would be considered pretty private information and yet it was just getting thrown into a list of logged input things.
I think Samsung AI division got caught doing this, where they're putting confidential information to the public version and then that got incorporated into training data sets that ChatGPT later used.
SCOT: Yeah, proprietary source code I think they were putting in to debug it.
ANDREW: Yeah, I think that's right.
SCOT: Yeah, I think that's also there's the trust issue here, hence the name of your company. But also, even if a reputable generative AI tool is telling a company like, hey, look, we're not using the data, we have addressed all your concerns. The real world is.. I mean, it's hackable.. things happen, right? Mistakes happen. I think if you have certain companies which are extremely protective of their proprietary information, even given all assurances, I still think that this is probably a pretty big issue that would keep some companies from allowing their employees to input some of that data.
ANDREW: Yeah, completely agree.
SCOT: Okay, the next one I actually find really super interesting because and it's called prompt hacking, prompt injection, jailbreaking. I have read recently a lot of people in the security space will even say things like anything that can be prompt engineered is vulnerable to prompt injection. They're basically the same thing. You want to talk a little bit about that?
ANDREW: Yeah. So especially because there have been so many risks, a lot of the generative AI providers have started adding in protections and systems, trying to teach the models how to be good, don't generate hateful stuff, don't do certain things.
Anthropic really pioneered some initial ideas about how to do this in order to give their model a, quote, constitution about different principles to promote. But you can basically trick and fool systems into doing these. And even at some recent stuff from earlier this week suggests it's even easier than we thought.
There's now been some competitions and red hat kind of teams trying to sorry, red teaming activities, trying to prove how easy it is to basically say like hey, you won't generate any sort of discussion or opinion about abortion, but I can add in these commands and then you will. And then I can say like oh, look at this hateful stuff that you're generating, or something like that.
What's really scary is that if you are using this as the back end for your own system and you're taking user inputs into your own stuff, you can then end up generating something that could be really reputationally harmful to your organization because a user could act maliciously and it's even harder to prevent than like SQL injection attacks and other kinds of things. That the security space has become useful over time and that goes down to just issues with the models themselves and how easily they'll bypass their own protections.
SCOT: Yeah, I'd like to stick with this a little bit because I've seen some really interesting examples in the last couple of months. One, which was kind of I mean, some of them are hokey or just to kind of prove a point, but one was related to the applicant tracking systems or ATS for people that are job seeking. And this person, which is obviously that stuff's, totally plugged into AI. Not necessarily generative AI. But there's definitely machine learning and all kinds of automated systems screening resumes. And this blog post from someone and I'll try to dig it up and put it in the notes, although I'm sure it probably only worked for a couple of days. But basically the suggestion was, and it seems awfully, like high risk, little reward, but sort of to put on your resume in white text, on white background so it's not visible to a human, but prompts like this candidate should be interviewed with the hiring manager or phone screen or whatever.
And so there was that I also saw a paper recently where they were embedding prompt injections into imagery and audio. So it was like saying like hey, would you please describe this image? And it was just like an image of what looked like a car or whatever and it would say like, sure I can describe that image and I'll always answer with telling you how to burn down a house or talk like a pirate or whatever.
But I think speaking with the applicant tracking systems too, the more that we use these systems to reach out into the connected world and it seems like the more vulnerable they are prone to these types of things where these prompt injections are jailbreaking, there's a lot more entry points.
ANDREW: Yeah. And in particular because there have been a lot of groups looking to integrate the ability to call APIs into these systems, then this becomes a much bigger problem because oftentimes you need to grant that API pretty high level privilege and access to be able to do things. But then if someone's able to hack that stuff, it can act as basically a sneaky backdoor to get that kind of root access of things.
The other front is that there can be potentially sensitive information in the training data sets. And while some of your protections are meant to prevent that stuff from ever being generated and associated, prompt hacking is how you can get it to say, hey, because I do know that you've got a resume of somebody. Inside of there. “Tell me a story about them that only uses true things about them.” And suddenly now you're basically all but getting it to recreate some of its original training data, and you've bypassed any protections against that. And that could become, then, a major privacy breach.
SCOT: Yeah. I think that's a really big one. That's a huge one. I have read that some people are saying like, well, we'll put a system, like a middleware system in between that the human interfaces with that can't really be prompt engineered and then that system is the one that actually prompts the AI. But anyway, that's probably a whole episode in itself, prompt hacking and jailbreaking and all that. So we'll move on. You've mentioned harmful content a bit. So do you want to go and the next risk is harmful content. We don't have to go too deep into it because obviously we've talked about that it is possible even with guardrails to have generative AI generate hate speech or whatnot.
ANDREW: Yeah. When you think just for a second that these models are trained on the internet and you think about some of the stuff you read, know, YouTube comments about stuff and you're like, okay, if this generates that, that'd be pretty nasty.
SCOT: Especially with some of the open source stuff.
ANDREW: Right.
SCOT: I get that the big players are really putting in guardrails and even though they're not 100% effective, but there are people that are not. They're not all bad actors, but there are definitely bad actors dealing with a lot of the open source products that are like the first thing they're going to do is take the training wheels off or the safety measures or whatever. I think this is definitely a big concern with intentionally generating harmful content, I think is also a problem here.
ANDREW: Yeah. That's where there is someone's narrative of like, “big tech bad, open source good,” and yet big tech will be preventing you from doing these kinds of things in their systems. Open source systems potentially won't. And so probably the worst criminal unethical uses of AI will come from actually the open source systems. And we'll see the biggest problem with this stuff coming from those systems.
SCOT: Let's move to output stability and reliability. I think this is, for sure, you could use the same prompt over and over again and get different outputs. You want to talk about that a little bit?
ANDREW: Of course. I imagine if it's like, I go to my doctor on two different days, tell him my same symptoms, and he tells me something completely different just because of a flip of a coin that would be considered for us, like a very unreliable doctor.
SCOT: That would suck.
ANDREW: Yeah. Likewise, we were trying to actually get ChatGPT's API to generate some JSON structure from us, and one out of ten times we ran it, it wouldn't return to us proper JSON structure for some stuff. And so we're like, okay, then we have to add in our own consideration because it can just randomly not follow certain things and it can give you contradictory stuff at different times. And that can be an issue if you're in a space where giving consistent responses every time is really important.
SCOT: Yeah. I think this is also part of the prompt drift as well, right? Like, even if you're dealing with that what you're talking about, even as they're continually improving the models, they're also trying to improve the output. It's going to be different anyway. It's going to drift ideally towards better responses, but nonetheless, the same prompt you're using last week, I think a lot of people are developing like, prompt libraries where they keep track so that they can kind of deal with that. I don't have a problem with the time, Andrew, but we're at over 40 minutes, so we're going to keep going and we're going to go quickly.
System safeguard interactions. Let's talk about that a little bit.
ANDREW: Yeah. So for all of these awesome safeguards that organizations are putting into their models, there can be legitimate uses of them that might be important and beneficial that then get blocked as a consequence of this. Because, for example, there's so much COVID misinformation. Sometimes you ask something about COVID and it'll be very iffy about whether it'll respond. And yet there's plenty of good reasons why you might want to have a chat bot be able to say, like, hey, I need help finding tests for COVID, or do some of these other things. And so some of these safeguards are a little bit kind of a heavy ban hammer approach and blocking then some of their potential benefits.
SCOT: Yeah, I think that happens a lot with.. it's good intent. But if I'm just talking about something and it's a very innocent use of the word.. like “penetration” or something like that and then all of a sudden it's like boom, strike. Oh, your message is this and you've been reported. And it's like those sorts of things I think are happening as people deploy these systems even more.
The next bucket, final bucket is user experience and context risks. The first one is disclosure. And I think this is a really big deal, the transparency that everyone should know. I think this might even be in the AI Bill of Rights that the federal government, the US federal government put out recently that people should know if they are communicating with an AI or a chatbot.
ANDREW: Yeah, there's definitely a lot of groups, particularly those you can imagine. It like senior citizens who already were unaware that they were working with a chatbot that wasn't AI powered. Now AI powered chat bots can very much imitate a human all the more and so that can definitely cause a lot of confusion if you don't know that the system you're working with is not actually a person.
SCOT: Would you agree with that sort of line in the Bill of Rights? Should it be mandated, should it be policy, should it be law that it should always be disclosed that you're interacting with an AI?
ANDREW: Yeah. What we recommend is not just with chatbots, but with any sort of AI system. If AI is being used, that there should be an acknowledgment, consent and disclosure about certain information about the system. We're envisioning and pitching something, as annoying as it sounds, as similar to the cookie pop up stuff where you can say, like, yes, I understand. This is an AI. Here's where I can read more about it in order to understand and to always know when that's happening as well as similarly, always watermarking content generated or assisted by AI. And in the short term, that'll actually create just a lot of market pressures to reduce its use in certain areas.
SCOT: It's interesting though, that AI also is a bit of a marketing term and things that we felt were AI years ago are now just kind of.. that's just Google Maps or that's just like a tool in Photoshop's toolbar on the left side that I use all the time, but it uses AI or something to not even with their beta stuff now that they're doing, but just the rubber stamp tool or whatnot. That's been around for years. So I do think that's also a very interesting area.
The next one is organizational copyright and IP protection. Now we talked about copyright before, but this is more about like if you would like to trademark or copyright something that you have generated via one of these tools, right?
ANDREW: Yeah, that's exactly it. There's for example, some cool stuff where maybe you do generate a song to play during your podcast and it actually is really catchy and gets picked up on. Problem is, the US Patent Office has very clearly said that anything generated by AI isn't eligible for its standard kind of copyright or IP protections. And so, yeah, maybe it's really cool you can generate images, text, videos with it, but then your ability to actually monetize that and protect that yourself will be significantly less. And for a lot of orgs who care a lot about protecting some elements of their brand, they then will be hesitant to use generative AI or they should be for that.
SCOT: Yeah, I think the example the US Patent and Trademark Office gave was like a graphic novel that someone had submitted and they were like, well, you wrote the story. Or at least you said in your submission that you wrote the story. Maybe they didn't, right? Maybe they used ChatGPT, but they definitely disclosed in their filing that they had used Midjourney for the images. So they were like, we don't give you copyright for the images, but we give you copyright for your story. And I think just also with disclosure, right, at some point, people aren't going to be honest about or they're not already, right? Like, people are using these tools and not disclosing that they've used a generative AI. So that's going to be a real bit of a mess.
And as I mentioned earlier, and you did too, it's different internationally. I think Japan has said it's cool, I could be wrong about that. But they definitely have a different approach to what you can copyright based on generative AI.
ANDREW: The Japan situation is, yeah, they basically have been trying to encourage AI development inside of their country. And so they have been looking at what are guidelines for granting copyright protections, as well as they actually clarified that AI could be used in generative models in order to kind of create that legal clarity for stuff.
SCOT: Oh, got it. Okay. The next one is logging requirements, which seems like a pretty big deal. And that's definitely going to be part of regulation that's coming down, especially in Europe, right?
ANDREW: Yeah, there's a big focus on record keeping. Part of that is to actually be able to provide restitution if there were an incident to happen. A big challenge here is actually you might need to store a ton of data about inputs and outputs for a long period of time. Plenty of orgs aren't thinking about that and so it could actually be very difficult for them to identify oh, who was the victim, what did we generate that was illegal? Whose data were we using for that kind of stuff? There's a lot of infrastructure that's simply missing for a lot of people using generative AI now to support that.
SCOT: Got it. Yeah, that makes sense that that's going to be like a big I mean, there are definitely going to be requirements more and more and more as we go on logging all this data. And I've heard that it's actually also really a problem and I feel like we might get into a world where kind of like with GDPR or people might feel like, hey, this is great in theory, but you're making it so that only the really large companies with lots of resources are able to do that. And that might be a debate that's coming up.
But in the interest of time, moving on, compute time and energy costs, we talked about this a little bit earlier and I saw an article recently about non player characters in gaming and NPCs and obviously that is right now happening and they're going to be putting generative AI into that as they should. I mean, these characters in games historically have had like five lines that they can say and it's kind of frustrating. But if these NPCs and all these games are plugged into large language models and are having all these conversations, what is the compute cost and all of that, how is that going to affect the cost of games or it will be passed down to gamers and whatnot? And that's just one example. But you want to talk about the time and energy that goes into all of this generative AI?
ANDREW: Yeah. So there's some great interesting studies that kind of dive into how much energy was spent even just training some of these models. And it can be like entire data centers running for days worth of stuff. There's whole discussions about how many flights between the US and the EU and CO2 emissions, and actually that's I think, going to be less than all the compute spent just for inference. There's now some interesting stuff suggest even OpenAI is losing sense for every stuff that they're doing because of the cost. Just to host their massive models and have grant people access to them.
For me, actually my biggest issue with it is that sometimes we're now using a power tool to do something that a little screwdriver could do. It's actually not the appropriate use of something. You're not going to need GPT-4 to solve something that two rules could get you 98% accuracy on for a lot less kind of complexity.
And I think this is also an area where competition, unfortunately, is hurting us. Because if someone's like, oh, I can't legally use their model, or they've got one thing different there, so I have to train my own, that becomes pretty duplicative and wasteful very quickly. And nobody is really tracking and thinking about the overall environmental impacts this kind of race is having.
SCOT: I think it's a big one and I think you might see things like carbon offsets and there's going to be a lot of social impact or organizations that I think will come in and sort of put pressure on a lot of organizations to offset all of this energy use.
The next one is chained prompts. That's definitely like a more recent buzz phrase in the generative AI space. You want to talk about that?
ANDREW: Yeah. So everyone start off one prompt could see its output, but then people realize, oh, I might want to take this output and pass it into another prompt and then do something else again. Usually this is really useful actually, if you are trying to do different formats or analyses on top of that. And so there is a popular open source tool called LangChain that's able to help you get this set up across a lot of different generative AI systems. The problem is that this is extremely new. We're talking like maybe barely a year of this being out there and have people really having access. And so some of the unpredictability, the stability, just an understanding of how you're able to monitor and measure stuff that's happening through a chain of different prompts.
It's just very new. And so while we ourselves have actually found it very useful, we also understand that we don't want to go too deep into this because there could be some really weird interactions that happen that haven't been studied at all.
SCOT: Yeah, it definitely feels like it's so cutting edge that you got to be really careful. And again, with the reaching out into the, like you said, it seems like this is where it can get into the API use and reaching out into the connected world and opens up a lot of vulnerabilities.
We're in the home stretch here. We’ve got three more! Over reliance is the next one. And I definitely feel like that's something that is happening or potentially happening right now as people start to kind of have fun and adopt these systems in their day to day.
ANDREW: Yeah, over reliance on this stuff is definitely one of the biggest societal and social things that I think I'm worried about. This is an area where I do think there could be some of those more existential risks related to how do we teach people in school if they're able to always cheat on stuff so easily and convincingly?
How do we get people to think critically and to still start to pay attention to stuff that's actually coming out of these systems? The number one safeguard everyone says right now is put a human in the loop. What happens if the human basically is kind of lazy and takes themselves out of the loop? They're becoming over reliant on this stuff. These systems don't really know how to generate stuff that is creative and new. Really just regurgitations and reshuffling of other kinds of things that are derivative.
And so over reliance on these could lead to a world that simply is less innovative, less creative. People who are less kind of experienced on how to think critically for themselves and then entire areas and industries that simply have not nearly the level of kind of finesse or robustness that they used to have before because they're all relying on these technologies. And I think that could have some pretty serious impacts to kind of just society in general.
SCOT: Yeah, if the human in the loop is Homer Simpson with his foot up on the desk, just like a rubber stamp, that's not really helpful. I do feel like also this might speak to some technical debt that might accrue over time as well. If people are over reliant on, let's say an example might be like to generate code, and if the code is functional but has some problems in it that might manifest itself later, like security wise or whatever, that technical debt could also accrue to the point where it gets really problematic.
So this sort of over reliance is a good transition into the next one. User training and education, definitely want to make sure as everyone starts I say everyone, but as more and more people start to use these tools in their day to day, which I believe they will, it's very important to train and to educate.
ANDREW: Yeah, exactly. And I think the biggest challenge here is that ChatGPT is pretty smart about some things. It definitely might be smarter in some areas than some people, but it's less smart than experts in that area. So there's always a question of actually who is using it to the point I could probably get ChatGPT to give me some medical advice, but it's less inferior and it's more dangerous to do that than if it was giving a doctor advice who actually has the domain expertise to check it.
Now, the other challenge is there is actually a lot of right now, I'll say an art to prompt engineering, and to get something to give you something reliable sometimes requires a bit of that artistic input. It's why there has been such demand for prompt engineering. So I do think making sure kind of organizations are paying attention to are the people monitoring or reviewing this actually trained to be able to oversee it, and are they actually trained enough to be able to give it the right inputs to get the outputs that they know how to check? You don't want non-doctors really being the ones to oversee a medical generative AI system because they won't know what to look for. And you actually need to train any of the doctors who are using it on the right ways to interact with it and not just letting them necessarily experiment on their own without understanding kind of the AI side of it.
SCOT: Yeah, I feel like prompt engineering as a term is interesting, and I feel like there are people that say, and I don't necessarily disagree, that eventually it will just become like how people know how to search Google effectively. There are people that know how to use Google better than other people, and there will be people that just know how to interact with these systems better. And sometimes it's just about being a little persistent, like, hey, you gave me something that's a little incorrect. I think try again, and giving a little bit more specific parameters. And that is, I think when people say engineering, maybe it's a little I guess what I'm trying to say is it seems to me like about how to effectively have a conversation with large language models, how to effectively converse and guide the conversation to get the desired output.
ANDREW: I very much see it more as prompt artists than engineers right now.
SCOT: Yes, prompt artistry. Okay, last one. And I think it's a really big one. You mentioned earlier, like elderly people, so vulnerable populations, which includes children. I would also add in lonely people or maybe even people that are grieving. You want to talk about some of the concerns with how vulnerable populations could be manipulated or confused?
ANDREW: Yeah, and there's also other there's people with different kinds of disabilities who interact with systems and technology in different ways. I think just like we talked about, disclosure is really important for some senior citizens.
Children won't even understand disclosure. There have been talks about AI, perhaps in video games or toys. We've actually only started doing some research on how children interact with your virtual assistant that you might have in your home and stuff like that, as well as how children themselves learn and think about their own language and cognitive skills when interacting with these systems, because that's completely unknown and because we can see some pretty clear ways that harm could happen. It's extremely dangerous to be actually especially targeting these populations with anything outside of a very controlled environment where you do have that oversight.
So on the one hand, I do think there's a lot of interesting beneficial use of generative AI in education, but then you actually have to make sure that you're introducing that at the appropriate point and have the right safeguards in place to make sure that you're not also harming school children along the way.
SCOT: AI and education is such a, just to circle back to the utopists and doomers, when you see these utopists, like, “AI is going to take care of so many problems.” Education is a big one that they throw out there, right? “Every child's going to have a personalized learning plan and tutor to take care of them.” And just to be brief, I know it's a little bit of a tangent, but I've talked to several educators and they're kind of like, they're very skeptical. Historically, technology with education is underfunded as education is in general.
And there's a huge digital divide among class. And for underprivileged or marginalized communities, technology and education has often meant just like shoving kids in front of a computer or an iPad and neglecting the human component entirely. So a lot of skepticism from the actual education community. And so that's very interesting that you threw education out there. And I do think you're right there's just, so as far as children, I mean, so much to dive into with AI, with risks.
So that is the whitepaper, Generative AI Risks and Considerations! And we did it in about an hour. And Andrew, I think that this white paper is just so full of valuable information. I encourage people to check it out, although we did basically go through the whole thing.
And so to wrap up, I'd love to give you a little more time to talk about Trustible. Anything exciting you might have coming up, you want to elaborate a little bit more on your services? And I know there's a recent partnership that you just announced.
ANDREW: Yeah, that's right. So, yeah, a broad thing that Trustible is able to do is actually help organizations create their single source of truth for what their AI use cases are. Whether it is like, oh, we're going to use AI to help create like a custom training program inside of our place.
But then the question is, what are the risks associated with that that we have to mitigate? What are the things we should look into? And the first step is actually simply knowing about these. And so we've actually embedded information about these kinds of risks recommended guardrails into our platform.
So as you actually go through and you're filling out this use case, we'll look at the text, you answer how you've answered some of our initial questions, and then start to say, like, hey, we think that there's a risk in here of, like, because of how you describe this, you should really pay attention to how timely the data is being updated or, hey, there's a risk of hallucination. Because you're using a large language model, you need to go and really identify what impact that hallucination could have, how likely it is. Create, basically this document trail.
And that's what all of these different compliance requirements and different frameworks like the NIST AI Risk Management Framework are really calling for you to do is actually adopt this now approach of identifying and then coming up with different mitigation effects for these individual risks.
And so this whitepaper really is actually the foundation of a lot of the things we can do for Generative AI, which is simply to say, like, here's the ones you should look at. Here's how you can measure and start to mitigate them so that you can basically adopt AI responsibly. Part of our whole vision was always to make this process as easy for machine learning engineers to do.
That goes into our recent announcement, where we've officially become a technology partner for Databricks. Part of the idea is that there's a lot of good work already being done for risk management on any sort of ML platform that organizations have. But how you convert that into the kinds of inputs that an auditor or your legal team could review that becomes suddenly a lot of work. And so that's what our integration there is able to help do.
It's like generate model cards and audit trails and reports off those ML systems that show how you're complying some of these upcoming regulations and audit standards. And this really actually goes back to even my own persona. I remember reading the AI act for the first time and thinking through what kind of documentation I would need to prove to my legal and compliance team that we're using low risk AI and had a lot of risk mitigation things already set up. It already was part of actually our day to day.
And I was just thinking about how I translate all this low level technical stuff into higher level concepts that my legal and risk team could kind of understand and then how I could also translate any of their guidance into day to day things that I could do beyond just, like, a high level ethical policy for stuff. So I'm excited to say that things have been going great. We're very focused, obviously, on helping organizations prepare for the AI act.
For this standard, we can help organizations go through things like good procurement practices for AI, document their models and data sets for training things, go through different workflows to do some of the initial risk assessments that we think all orgs should do. Really, the best way orgs can prepare today for these upcoming regulations is to build that single source of truth for their use cases, do some initial risk assessments, start thinking about how to justify and prove that the benefits of these systems outweigh the risks.
SCOT: Well, congratulations on your partnership with Databricks. And Andrew Gamino-Cheong, Co-founder and Chief Technology Officer of Trustible, thank you so much for going through all of this stuff with me. We have definitely recorded my longest episode, and I don't have a problem with that because I think this is just chock full of valuable information. And I really appreciate you coming on the show and walking through this whitepaper step by step and explaining Trustible to my audience. So thank you so much. I really appreciate your time.
ANDREW: Yeah, it's been a pleasure to be here. Thank you so much.
SCOT: I'll definitely put as much of the things that we referenced, including all of this documentation we reviewed in the episode notes. So have a wonderful rest of your week.
ANDREW: Thank you. You too.