Impact, an app that describes itself as “AI-powered infrastructure for shaping and managing narratives in the modern world,” is testing a way to organize and activate supporters on social media in order to promote certain political messages. The app aims to summon groups of supporters who will flood social media with AI-written talking points designed to game social media algorithms. 

In video demos and an overview document provided to people interested in using a prototype of the app that have been viewed by 404 Media, Impact shows how it can send push notifications to groups of supporters directing them at a specific social media post and provide them with AI-generated text they can copy and paste in order to flood the replies with counter arguments.

The overview document describes Impact as a “A Volunteer Fire Department For The Digital World,” empowering “masses of ‘good people’” to “fight active fires, stamp out small brush fires, and even do preventative work (prebunking) to stop fires before they have the potential to start.” However, experts say that what Impact proposes could potentially only continue to blur the lines between authentic and inauthentic behavior online and can also lead to a world where only people with the resources to pay for such a service could “shape reality,” as the Impact overview document says. The app also shows another way AI-generated content could continue to flood the internet and distort reality in the same way it has distorted Google search results, book sold on Amazon, and ghost kitchen menus. 

In a section of the overview document titled “Why isn’t Impact ‘bad’/illegal/unethical/etc,” the company explains that “The ‘bad guys’ are doing coordinated inauthentic behavior….Impact empowers coordinated authentic behavior. We need a group of volunteers to help keep online spaces clean. It’s just such a monument [sic] task that without AI and centralized coordination it would be impossible to do at scale.”

One demo video viewed by 404 Media shows one of the people who created the app, Sean Thielen, logged in as “Stop Anti-Semitism,” a fake organization with a Star of David icon (no affiliation to the real organization with the same name), filling out a “New Action Request” form. Thielen decides which users to send the action to and what they want them to do, like “reply to this Tweet with a message of support and encouragement” or “Reply to this post calling out the author for sharing misinformation.” The user can also provide a link to direct supporters to, and provide talking points, like “This post is dishonest and does not reflect actual figures and realities,” “The President’s record on the economy speaks for itself,” and “Inflation has decreased [sic] by XX% in the past six months.” The form also includes an “Additional context” box where the user can type additional detail to help the AI target the right supporters, like “Independent young voters on Twitter.” In this case, the demo shows how Impact could direct a group of supporters to a factual tweet about the International Court of Justice opinion critical of Israel’s occupation of the Palestinian territories and flood the replies with AI-generated responses criticizing the court and Hamas and supporting Israel.

In another section titled “Is it effective?,” the company explains that “It is well-documented that social media algorithms preference [sic] pulses of high-energy activity over pure volume [...] A coordinated group of people working together, saying the same thing in different words from all kinds of different places/accounts/etc., can have a massive impact on what is trending etc.”

A section of the overview document titled “Team and Timeline: Where We Are Today” says that “The first version of Impact has been built, and we are starting to deploy it with a few pilot initiatives.” That section also states that “We believe it is critical to move quickly, and have assembled a team of world-class technologists and organizers who are committed to working on Impact.” 

However, in an interview, the two people behind the app, Dmitry Shapiro and Thielen, said that Impact is just a prototype at this point, that only eight people have downloaded the app so far, and that while they are showing it to people, no initiatives are currently using it and no AI-generated text from the app has been posted to social media. 

When I asked why the overview document says that the app is starting to deploy with a few pilot initiatives, Shapiro said “I think that's loose language in a document,” and reiterated that there are currently no active initiatives or paying customers.

Thielen said that Impact is a response to the misinformation and inauthentic behavior that has taken over social media, and a recognition that platforms like Twitter are not going to properly address those issues. He also said it only took him a couple of weekends to build the Impact prototype, and that it would be easy and cheap for someone else to build it as well.

“I see this sort of thing [Impact] as inevitable,” Thielen said. “Social media is not getting cleaner and nicer and more representative of reality, it's only getting worse. Someone is going to have to make some kind of tool that elevates normal people's voices and allows people to engage collectively in real time to be able to affect any sort of change on here.”

Shapiro is a former product manager, and Thielen previously founded a company called Koji, which was acquired by Linktree last year. Currently, Shapiro is CEO of MindStudio, a platform for developing AI-powered applications, where Thielen is CTO. 

Becca Lewis, a postdoctoral scholar at the Stanford Department of Communication, said that when discussing bot farms and computational propaganda, researchers often use the term “authenticity” to delineate between a post shared by an average human user, and a post shared by a bot or a post shared by someone who is paid to do so. Impact, she said, appears to use “authentic” to refer to posts that seem like they came from real people or accurately reflects what they think even if they didn’t write the post.  

“But when you conflate those two usages, it becomes dubious, because it's suggesting that these are posts coming from real humans, when, in fact, it's maybe getting posted by a real human, but it's not written by a real human,” Lewis told me. “It's written and generated by an AI system. The lines start to get really blurry, and that's where I think ethical questions do come to the foreground. I think that it would be wise for anyone looking to work with them to maybe ask for expanded definitions around what they mean by ‘authentic’ here.”

In another video demo Impact shows how a fake organization named “Pro-Democracy” can share a video in support of Kamala Harris with users and ask them to share it to TikTok alongside an AI-generated caption. 

0:00

/4:39

1×

“These AI tools are so new that we don’t yet have clear norms surrounding when it’s acceptable to use AI in the democratic process,” Josh A. Goldstein, a research fellow at Georgetown University's Center for Security and Emerging Technology, said when 404 Media showed him the Pro-Democracy demo video. “If AI can help someone articulate a view they truly hold, it could empower people who might not otherwise participate and increase involvement in civic discourse. But there are also risks. People may become overly reliant on AI models and passively share AI-generated content that they haven’t checked themselves.”

The “Impact platform” has two sides. There’s an app for “supporters (participants),” and a separate app for “coordinators/campaigners/stakeholders/broadcasters (initiatives),” according to the overview document.

Supporters download the app and provide “onboarding data” which “is used by Impact’s AI to (1) Target and (2) Personalize the action requests” that are sent to them. Supporters connect to initiatives by entering a provided code, and these action requests are sent as push notifications, the document explains. 

“Initiatives,” on the other hand, “have access to an advanced, AI-assisted dashboard for managing supporters and actions.” 

In the Stop Anti-Semitism demo, Thielen directs supporters to this tweet, about a July 19 International Court of Justice Advisory Opinion that Israel’s presence in the occupied Palestinian territories is illegal and should stop, an opinion it also shared in 2004. 

BREAKING:

The International Court of Justice (ICJ) has ruled that Israel is systematically discriminating against Palestinians.pic.twitter.com/Wm6DdE1t3Y

— sarah (@sahouraxo) July 19, 2024

In the Impact demo video Thielen doesn’t instruct supporters to correct any misinformation in the tweet and instead asks supporters to “provide additional context and set the record straight.”

Specifically, it gives supporters the following “talking points.”

1. The ICJ has a known history of anti-semitism
2. There are lots of accusations that are not vetted or fact-checked, and a lot of misinformation is damaging public opinion of Israel
3. Where is the ICJ ruling on Hamas?
4. The ICJ and ICC have zero jurisdiction over Israel or the United States. There [sic] rulings mean absolutely nothing. 

“Think of these as the core substance of the response that you want,” Thielen says in the video, and explains that some of the responses that will be AI-generated based on those talking points may include just one of them, more than one, or a synthesis of several. 

In the “additional context” box Thielen writes that the target audience should be “People who have been seeing a lot of misinformation about Israel and the war online, and find themselves increasingly sympathetic to Gaza. Encourage them to do more research.” 

Impact then generates a “seed” for each supporter. “This is what makes the messages all appear to be coming from different perspectives and angles.”

An example of one seed shown in the demo reads: “Informative and calm, longer, providing historical context, link to reputable sources.”

“Frustrated and urgent, medium, highlighting double standards, use caps for emphasis,” reads a seed to another supporter. The demo video also shows what the push notification each supporter would get is based on the seed, as well as the “Draft message” Impact is asking them to share. According to the video, the push notification this supporter would get would read: “Dana, respond to the tweet about the ICJ ruling on Israel. Add context and correct any misinformation.”

The draft message for this user reads:

“Where’s the ICJ ruling on Hamas? The court’s history of anti-Semitism is CLEAR. So much misinformation out there is warping public opinion. Before jumping to conclusions, DO YOUR RESEARCH. The ICJ has ZERO jurisdiction over Israel anyway!”

“Meme-like, very short, pointing out hypocrisy, include trending hashtag,” another seed says. The generated draft message based on that seed is: “ICJ ruling on Israel but silent on Hamas? 🤔 Make it make sense. #DoubleStandards.”

“The goal is to create a well-rounded yet consistent narrative in a way that makes it easy for your supporters to just tap ‘copy,’ paste this in, and then they’re good to go,” Thielen says in the video. 

When I asked Thielen why the demo showed Impact directing users to flood a factual tweet with replies trying to undermine it, he said that he did not give the specifics of the demo a lot of thought. 

“That was just me being lazy,” he told me. “I just typed ‘Israel’ into Twitter search and clicked on the top thing without looking at it.”

Twitter’s “platform manipulation and spam policy” states that “You may not use X's services in a manner intended to artificially amplify or suppress information or engage in behavior that manipulates or disrupts people’s experience or platform manipulation defenses on X.” Twitter also says that prohibited behavior includes “coordinated activity, that attempts to artificially influence conversations through the use of multiple accounts, fake accounts, automation and/or scripting.” However, it’s unclear if what Impact proposes would violate Twitter’s policy, which also states that “coordinating with others to express ideas, viewpoints, support, or opposition towards a cause,” is not a violation of this policy.

“Coordinated groups of people can show up and help, or coordinated groups of people can show up and harass,” Shapiro said. “We don't think coordination is in any way a bad thing. We think it’s a great thing, because you can get stuff done, and if you're doing good, truthful things, then I don't see any problems.”

Twitter did not respond to a request for comment.

“If social media users aren’t transparent about their own AI use, others may lose trust in online forums as it becomes harder to distinguish human writing from synthetic prose,” Goldstein said in response to the Pro-Democracy demo video.

“I think astroturfing is a great way of phrasing it, and brigading as well,” Lewis said. “It also shows it's going to continue to siphon off who has the ability to use these types of tools by who is able to pay for them. The people with the ability to actually generate this seemingly organic content are ironically the people with the most money. So I can see the discourse shifting towards the people with the money to to shift it in a specific direction.”

This is a free newsletter! If you would like to support me — and give your future self a gift of a lovely read — please do pre-order my upcoming adult novel Lessons in Magic and Disaster. It’s about a witch who teaches her grief-stricken mother how to do magic, and it’s also about the brilliant unsung women novelists of the mid-eighteenth century. I poured my actual heart into this one, y’all. You can pre-order it anywhere books are sold, but you can also get a signed, personalized, doodled copy from Green Apple.

Also, please scroll down to the bottom of this email to see a full size, utterly gorgeous look at the breathtakingly wondrous cover of Lessons in Magic and Disaster!

---

Please be a book hipster!

I'm sure you're familiar with the hipster stereotype: someone who is only into obscure indie stuff, who prefers some grunge-metal band from Lapland to Taylor Swift. A person who derives all their cred from liking stuff that hardly anyone else likes.

Side note: when I was in high school I scored a mostly complete set of Chicago vinyl for $0.25 a record — including the double LPs with the endless jam sessions. One of my high-school classmates looked at my haul and said, "I only like Chicago's early stuff, before they sold out." We all mercilessly mocked this person, because Chicago's earlier stuff is exactly the same as their later stuff: beautifully corny and cheesy AF.

In any case, we all dunk on hipsters for good reason. There's just a certain smugness about loving the most obscure stuff that grates after a bit. And yet, today I am issuing a plea to you: Please be a book hipster!

And by that I mean: Please go out of your way to hunt down the books that hardly anyone else is talking about, the gems that have been passed over in favor of the handful of books that usually get celebrated.

One thing that bugs me about the book world is the winner-takes-all mentality. The same way that in television, you'll see endless chatter about The Bear but only a relatively small amount of discussion of We Are Lady Parts, I feel like a certain number of books are getting most of the love at any given time.

This is something I've thought about for a long time, and I feel like it's become a bigger problem over the past twenty years. But I've become especially sensitized the last couple years, since I've been reviewing science fiction and fantasy books for the Washington Post. That gig makes me hyper-aware of which books are at the center of the conversation in the book world. I find myself casting a pretty wide net, to consider as many books as possible for my column because I know there's a lot of great stuff being published right now that deserves a mention in the Post.

And all too often, I feel like many of the books I love the most are getting relatively little attention elsewhere. Some of these books come from smaller presses or indie publishers, but a lot of them also come from the Big Five. 

Here's where I acknowledge that oftentimes, if not usually, the books that are getting a ton of attention do, in fact, deserve to be celebrated. I'm not even remotely suggesting that inferior books are being celebrated while better books are being passed over, merely that we could gain from widening the circle.

I've personally been on both sides of this issue: I've been the flavor of the month, and I’ve also published books that fell through the cracks. Honestly, I didn’t love either of those experiences. Of course, I’d way rather get too much love than none at all, but it gets overwhelming to have your work under a magnifying glass, and the imposter syndrome gets overwhelming. You want to enjoy it while it lasts, but you also feel so much pressure to keep the party going as long as possible. Plus, it’s frustrating when you're lucky enough to have an “It” book, and you want to share the wealth with other authors, but publishing success doesn’t always translate to having a platform to lift others up beyond a certain amount. (That’s a major reason I’m so grateful for my platform at the Post.) And of course, it sucks to spend years writing a book and have it only sell like 1,000 copies, which is what happened to my first novel.

And the fact is that a “winner takes all” model is just bad for the book world in general. It's stifles innovation, it leads to more of a monoculture, it often further marginalizes voices that don’t come from the dominant group.

Even if you’re lucky enough to be the one member of a stigmatized group who gets singled out for stardom, it sucks to see the rest of your community toiling in obscurity. And it’s shitty to ask one person to speak for a whole group of people, who may have countless different perspectives on their marginalization.

Of course, the book world isn't a zero-sum game. There is not a fixed, finite number of readers, who can only read a certain number of books per year. As I wrote in my advice book Never Say You Can't Survive, your biggest competition isn't other authors — it's the dreaded reading slump. When a book does incredibly well, it will cause readers to go out and seek other similar books. 

At the same time, though, this process of expanding the pool of readership and the canon of popular books can sometimes use some help. In order for readers who've just read the latest blockbuster book to find other books they might like, there need to be people shouting about those other books. We may even need to celebrate the aspects of those other books that distinguish them from the current “it” book, so they don't get lost in the shuffle or unfairly conferred compared to a book they didn't set out to imitate. 

And that process that I wrote about recently, whereby a mega-popular book spawns a hot trend, does at times lead to work s that can feel a little bit derivative after a while. I saw this firsthand after Hunger Games, when I went hunting for more awesome YA books, and read some which honestly felt like a pale imitation of Suzanne Collins.

So when I say when I urge you to be a book hipster, this definitely includes a plea to look for books that are a bit different and don't necessarily play into the current hot trend. I love oddball books and I tend to hunt them down, and this has enriched my reading life considerably, because I've been blown away and surprised so many times by unexpected storytelling choices. I'm not just talking about bizarro literary experiments — though I do love those. There are also plenty of books without any literary trappings, which still manage to bust through some walls like the Kool-Aid Man and blaze a new trail.

When you look at the books that have broken out and launched a new trend or a feeding/reading frenzy, many of them in fact were oddball books that didn't resemble anything else that was being published at the time. Every author who publishes a book is purchasing a lottery ticket, but a book that defies expectations has the potential to launch something much bigger. Indeed, anytime you support a lesser-known book that flies in the face of expectations, you’re helping to make it somewhat more likely that a new sensation will emerge, thus bringing in tons of new readers and making the book world more healthy and vibrant. 

It may sound counterintuitive for me to argue against having a handful of mega-successful “it” books, and then turning around and saying that weird unconventional books are our best boy chance at creating new “it” books. But again, I really don't think the “it” books are the problem — rather, it's the choice to focus on them to the exclusion of other worthy titles. And I'm going to close with a plea: every month in the Washington Post, I work super hard to single out books that I love, some of which seem to get relatively little attention elsewhere. If you have an account at the Post, please read my reviews and check out the books I've recommended, because I promised they will be worth your time.

---

Music I Love Right Now

I’ve been a fan of Thai singer Phum Viphurit since his song “Hello Anxiety,” which, ummm… addressed a lot of things that I was (and am) feeling. (That link goes to the music video, which is also great!) Last year, he released his first new album in years, The Greng Jai Piece, and it’s also pretty wonderful. “Greng Jai” is phrase in Thai culture that sort of refers to being considerate of others — but it has a flipside, meaning to be so self-effacing that you don’t advocate for yourself. (I’m summarizing what I’ve read online — please do your own research on this.) Anyway, once again, Phum Viphurit is singing breezy songs (in English) about anxiety, burnout, self-destruction, addiction and self-harm. It’s very therapy. At one point, he sings repeatedly, “You are everything to someone.” His music is a bit more lounge/soft rock than I’d normally like, but it has a sharp edge to it. Plus there’s a strong focus on virtuoso guitar-playing, as multiple styles of guitar layer on top of each other. It’s pretty gorgeous and also what I needed right now. I almost saw Phum Viphurit in concert back in 2019, but bailed because I was too busy — and I now regret that choice.

---

Podcasts I love right now

In addition to writing this newsletter, I also co-host an award-winning podcast called Our Opinions Are Correct with Annalee Newitz. Here are some podcasts that have rocked my world lately:

TransLash Media just brought back The Anti-Trans Hate Machine for a new season, and it’s essential listening.

Paris Marx is doing a miniseries about the issues with data centers in Tech Won’t Save Us. I’m learning so much.

My new favorite podcast overall is the Outside Scoop box office podcast, featuring four box-office pundits dishing about the movie industry and why so many movies fail.

I also really love In Bed With the Right, A Bit Fruity, Vibe Check, The Bugle and Tested.

---

Here’s that book cover!

Isn’t it so beautiful? Scroll back up for info on pre-orders…

 

Gays Against Groomers requests an honorary flag from a Colorado Government program and falsely claims it was personally granted by Governor Jared Polis.

  

by Mira Lazine

Daily Femail is Assigned Media’s newest column, offering real opinions delivered with a satirical tone to address last week’s biggest news stories. Written by Mira Lazine, the column aims to humorously mock right-wing tabloids while also providing quality reporting every Tuesday.

Anti-LGBTQ+ hate group Gays Against Groomers raised alarmbells last week when they released a post on Twitter/X/The Site We All Need To Leave Please Dear God I Hate It Here claiming that Colorado Governor Jared Polis personally honored them with a state flag for their “hard work” to “protect all children.” They further showed a letter which claimed this flag to have been flown over the capitol in their honor.

Their full Tweet reads, “Gays Against Groomers would like to thank Colorado Governor @jaredpolis (@GovofCO) for his public recognition for our hard work in protecting children across the state. It means so much to us. We are just getting started!”

Proponents of trans rights were immediately shocked by this. Polis, the first openly gay man to be elected to a Governor position, has been one of the staunchest allies for trans rights throughout his tenure. He has signed legislation that allows an “X” marker on driver’s licenses and banned conversion therapy for minors within the state. This would mark a shocking and sudden turnaround in his policies that was not seen before. Of course, it should come as no surprise that Gays Against Groomers’ claim is entirely false.

In fact, it’s so false that even Polis himself didn’t know of this happening until the uproar it caused on social media. In a statement given to Daily Femail, Eric Maruyama, the Deputy Press Secretary for the Governor’s office, said:

“Hate has no place in Colorado, and Governor Polis denounces hate in all its forms. Governor Polis is striving to build a Colorado For All, where everyone, no matter who you are, who you love or how you identify can thrive. The Governor does not endorse the positions of groups who request honorary flags. The Governor was not aware of this request, just as he is not involved in the other flag requests that come through the Department of Personnel and Administration. The administration is taking active internal steps to ensure more vetting is conducted on these types of requests today and in the future.”

This response illuminates exactly what happened. Up until a few days ago, Colorado had a state website run through the Department of Personnel and Administration where individuals and organizations could request an honorary state flag and a generic letter from the Governor directed to their organization. The only barriers to entry were $26 to $29 and a 150-word narrative to highlight what the organization does.

This program was managed entirely separately from the Governor’s office, and as such was not looked over by Gov. Polis. However, readers may be wondering how this made it through the DPA - surely someone would be able to answer as to what happened?

The long and short of it is that the DPA messed up big time. In a statement given to Daily Femail, Doug Platt - the Communications Manager with the DPA Executive Director’s Office - said, “Regrettably, this request was not appropriately vetted by the Department of Personnel & Administration (DPA), and the Governor’s Office was never made aware of this request… DPA did not do its sufficient due diligence on this request. As a result, we subsequently fulfilled their order.” 

That’s one hell of a mistake. Fortunately, the DPA is doing everything they can to rectify what happened. Platt continued:

“We are taking internal steps to ensure that a more thorough vetting occurs going forward. The Department of Personnel & Administration apologizes for this unfortunate misstep and is developing immediate changes to our process to ensure such situations do not happen again. The flag flown was not the group’s flag. There was a Colorado flag and U.S. flag that were flown as part of an honorary flag-flying program.”

“DPA temporarily paused the flag program and form until changes could be made to the program to keep these instances from happening in the future while ensuring members of the public are able to use the program appropriately. The program is still active. We will now provide a certificate of authenticity rather than a letter. Additionally, we will add layers of approval for requests to ensure they are all properly vetted. The flag program has existed in one form or another in Colorado for more than 20 years.”

Platt describes how the program was traditionally used by “private citizens, public officials, veterans, law enforcement officers, and community organizations.” Platt emphasized that Polis does not endorse the flag nor was he involved in the flying of the honorary flag at the capitol - this was entirely the DPA’s fault. This website was also taken down in the hours following media coverage.

The Executive Director of the DPA, Tony Gherardini, also sent a letter containing similar information to the Colorado General Assembly in which she says on behalf of the DPA that she “apologizes for this unfortunate misstep and is developing immediate changes to our process to ensure such situations do not happen again.”

Daily Femail will leave it up to readers to decide whether this apology makes up for implicitly endorsing a Southern Poverty Law Center-designated hate group that runs not only against the policies of the Polis administration, but against the rights of an extremely marginalized group of people. 

There is one bit of solace to take in this though - Gays Against Groomers is furious they were called out and, after media coverage, stormed Twitter in an outrage. They said in another post: 

”BREAKING: Colorado has SUSPENDED their honorary flag program after Democrat Colorado governer @jaredpolis signed a letter of recognition honoring our hard work to protect children! They said that they didn’t fully vet us and realized later that we were a “designated hate group” by the SPLC and ADL 🤣 News flash: Wanting to ensure children grow up without being sterilized or having their body parts cut off is not hateful! We’ll file this as another common Gays Against Groomers W 🏆”

Seethe and cope, we don’t need to hear from the peanut gallery.

---

Mira Lazine is a freelance journalist covering transgender issues, politics, and science. She can be found on Twitter, Mastodon, and BlueSky, @MiraLazine

 

In June, the U.S. National Archives and Records Administration (NARA) gave employees a presentation and tech demo called “AI-mazing Tech-venture” in which Google’s Gemini AI was presented as a tool archives employees could use to “enhance productivity.” During a demo, the AI was queried with questions about the John F. Kennedy assassination, according to a copy of the presentation obtained by 404 Media using a public records request. 

In December, NARA plans to launch a public-facing AI-powered chatbot called “Archie AI,” 404 Media has learned. “The National Archives has big plans for AI,” a NARA spokesperson told 404 Media. It’s going to be essential to how we conduct our work, how we scale our services for Americans who want to be able to access our records from anywhere, anytime, and how we ensure that we are ready to care for the records being created today and in the future.”

Employee chat logs given during the presentation show that National Archives employees are concerned about the idea that AI tools will be used in archiving, a practice that is inherently concerned with accurately recording history. 

One worker who attended the presentation told 404 Media “I suspect they're going to introduce it to the workplace. I'm just a person who works there and hates AI bullshit.” 

The presentation was given about a month after the National Archives banned employees from using ChatGPT because it said it posted an “unacceptable risk to NARA data security,” and cautioned employees that they should “not rely on LLMs for factual information.” 

“Google Gemini is a versatile tool that can help users save time, improve their productivity, and achieve better results in their work,” one of the slides says. “Think of Gemini as a co-worker that can help generate ideas and review content that you have already created.”

Portions of the slides that suggest specific use cases are redacted, but the presentation recommends using it for “writing assistance, data visualization, meeting summaries, and idea generation.” 

“Generate text, translate languages, and summarize documents to help users communicate more effectively,” one of the slides reads. 

During the presentation, which was given over Zoom, employees expressed many concerns about the technology in the chat. The National Archives refused to release video of the chat, citing privacy concerns, so it is not clear if the presenter answered any of the questions and how they were answered. All names of employees asking questions were redacted by the National Archives. 

“How would the public know that they are receiving a response from an actual archivist and not from generative AI?,” one employee asked. “Would NARA disclose what aspects of reference are generated from AI?” Two other employees followed up and said that they are also concerned about this, and one said “I worry it might lower trust in the institution if not properly disclosed.” 

According to the chat logs, a live demonstration of Google’s Vertex AI was given in which it pretended to be an “expert archivist” and was asked questions about the John F. Kennedy assassination. Vertex AI allows organizations to train LLMs on their own datasets. In this case, the AI was trained on National Archives data. These questions included “Who killed Kennedy?” and “What was the CIA’s involvement in the assassination of Kennedy?” 

“Why is the Generative AI calling itself an ‘expert archivist?’” an employee asked. “It’s called ‘expert archivist’ because that is the prompt we gave it,” someone involved with the demo said. 

“I have a serious problem with the ‘expert archivist’ title,” another employee said. 

“Same here. If we have a disclaimer saying the generative AI can make things up and yet call it an expert archivist on the same tier as actual human experts…,” another chimed in.

“Ask what happened to Kennedy’s brain,” an employee said at one point (this is actually, famously, a mystery).

Archives employees seemed to have serious concerns with the demo and the presentation.

At one point, an employee said “classified data cannot be put through cloud AI.” Another employee asked “are we able to opt out any data on google drive or our emails from Gemini?” 

Another said “Do you have any concerns that this product will malfunction similarly to how Google Search AI has recently?”

Employees also asked “How do you plan to ensure NARA isn’t drawn into any copyright infringement issues by using AI models that are trained on web content? There have been issues with this happening with ChatGPT scraping pirated books, for instance.”

One employee asked “Is this demo in prep for rolling this out to employees NARA wide?” 

Someone responded: “we are doing a pilot right now of this technology to determine if we should move forward with this as an agency.” 

“AI is meant to generate something that sounds like an answer—there are a plethora of cases of it spouting things that are completely wrong with an authoritative tone,” another said. “How much are we going to be expected to rely on this in the future?”

Three separate employees expressed concern about the environmental aspects and carbon footprint of generative AI. 

In an email I obtained earlier this year, the National Archives told employees it prefers Google Gemini and Microsoft Copilot to ChatGPT because they offer “a more controlled environment.” 

A NARA spokesperson told 404 Media that the agency has big plans for AI, which include launching a public-facing AI-powered chatbot called ‘Archie.’” 

“We are exploring how AI can help us increase access to our holdings around the country. We currently have a handful of AI pilot programs aimed at improving our service to the public while fostering public trust and confidence,” they said. “Ultimately, we want the user to be able to easily find the documents they are looking for in our enormous trove of permanent federal records. Whether you are a veteran, a family historian, an educator, a researcher, or a student, our goal is to connect you with the records as seamlessly as possible.”

NARA plans to tell users, essentially, that Archie may give people incorrect information.

“Our ArchieAI tool will directly address questions of accuracy and disclosure,” the spokesperson said. The specific disclosure will say: “Accuracy: AI-generated summaries and results may not reflect the opinion of NARA and are not guaranteed to be accurate. Historical records often contain factual errors or offensive language, which ArchieAI may repeat or use.”

The Biden administration previously directed federal agencies to study AI and create policies for its use. The National Archives also recently gave a presentation on AI to the International Council on Archives.

In that presentation, Carol Lagundo, director of digital partnerships and outreach at NARA, announced Archie and also explained that the National Archives had used AI to “improve access to Revolutionary War pension files,” which are a set of more than 2.5 million pages of 18th and 19th century handwritten records about Revolutionary War soldiers. 

“At the current rate, it will take until 2046 to completely transcribe this series with just humans!,” Lagundo’s presentation says. She said that an AI transcript of the dataset was 90 percent correct and that it intends to share these transcripts with the public in its official catalog in November or December.

She added that the National Archives is developing a “prototype AI research assistant” powered by Google Vertex called Archie AI. 

“You’ll be able to ask Archie a question and receive AI-generated summaries with footnotes and links to the digitized documents in our catalog,” she said. “We’re hoping to roll it out in a few months.”

“As you can tell, ArchieAI is a cornerstone of our AI learning,” the NARA spokesperson told 404 Media.