"The Most [Undecipherable] Difficulties": Sustaining and Transcribing the Papers of the War Department
One of the most complicated websites RRCHNM has been sustaining in recent years is the Papers of the War Department (PWD), a crowdsourced transcription project to recover and make accessible the once-lost papers of the War Department whose offices burned to the ground in 1800. The project began with document collection and digitization efforts in the 1990s, but only became a website and an RRCHNM project two decades ago, in 2006.
The PWD website was designed as an Omeka site, using the Scripto plug-in and MediaWiki to enable crowdsourced transcriptions of the documents. Unfortunately, wikis are a prime target for hackers and every time one got through the firewalls, our wiki filled up with the usual kind of internet spam for adult entertainment. Furthermore, while crowdsourcing promotes the illusion of work being done by others instead of a core project team, proper crowdsourcing requires significant oversight from a project team to ensure that contributions are legitimate and meet team standards. Like all soft-funded projects, however, this work cannot easily continue after funding comes to an end and after the project team members have departed to new jobs and institutions.
In 2026, as part of our ongoing sustainability work we determined that the time had come to migrate from Omeka + MediaWiki to a static site with a lower security risk profile, specifically using Hugo. See this blog post for more on our CMS to Hugo pipeline. Because only around 6% of the 42,000+ documents had been manually transcribed, we also wanted to increase the searchability of the site by expanding the number of available transcriptions. By 2025 it appeared Handwriting Text Recognition (HTR) with Large Language Models (LLMs or “AI” as the hype would have it) had reached a quality tipping point where legible digital surrogates of originally handwritten 18th century documents could be transformed into reasonably accurate transcriptions (Dan Cohen Nov 2025). We therefore determined to redesign the site so that each item displays both a human transcription (if available) and an AI transcription (as better than nothing) and embarked on a quest to mass-transcribe the digitized War Department documents using LLMs.
Our first step was to determine which LLM to use. We quickly narrowed our options down to Gemini Pro 3.1; Claude Opus 4.6; and Claude Sonnet 4.6. While Gemini produced the best results, it had a number of disadvantages at the time we scoped the project—most importantly, it was not yet possible to set token maximums to prevent runaway processes from spending ghastly amounts of money before you could stop it (this has since been added). Of the two Claude models, Opus produced slightly better results but Sonnet was faster and the most affordable option.
Given this project is completely unfunded at this point and all sustainability work comes at the expense of our active projects, we decided to primarily go with Sonnet 4.6 and only use Opus when the weekly Sonnet usage limit hit. This process began in April 2026 and completed in August 2026. There were occasional hiccups now and again—including rate limits, bugs in the script processing, and a mis-alignment of document IDs to images due to a bug in Omeka we had to fix—but all available images now include a machine transcription.
Instead of running our scripts through Claude’s batch API, we spent several months sending them in smaller chunks through a Claude Max subscription to minimize costs. Claude’s command line tooling provides the option of a -p flag, which allows us to run Claude non-interactively. What that means is rather than interact with Claude in its usual chatbot-like mode, we simply provided a prompt based off what had already been previously developed for human transcribers and the set of images per document, arranged in an image manifest we generated via the Omeka API. Giving non-interactive Claude the prompt and the images to transcribe meant we could script a way to send requests, which gave us back a JSON response that we appended into a large AI transcription JSON document. Once that document was fully compiled, we simply used Hugo’s data tooling to associate a transcription with a document’s unique identifier which gave us a public-facing UI for users to toggle between human or machine transcriptions.
As part of this effort we added two features on the website, a transcription dashboard to track what’s been transcribed, by what method, and what remains, and a brief document about our machine transcription methodology. About three quarters of the collection now includes transcriptions and the untranscribed items are primarily things like citations or references to texts that we do not have the images for. The process took a while, something we certainly could have made much shorter had we had the resources to pursue the bulk API route, but using LLMs meant this was still well within our resources working unfunded and around the edges of our other projects.
Our main takeaways from this sustainability and transcription experiment are threefold:
- LLMs make it significantly easier to migrate CMS-based sites to different platforms, including but not limited to Hugo, while maintaining original design and functionality
- we are not to the point where we can eliminate human review and intervention or the expertise of paleography, but these systems are good enough at transcribing documents that they can be used in tandem with traditional transcription methods
- even if you don’t have the resources to transcribe a historical document properly, the LLM (“AI”) transcripts are at least sufficient to give users a more robust access pathway for historical document images
We aren’t going to be doing away with manual transcription any time soon, but for our sustainability purposes and digital project backlist, LLMs can dramatically increase people’s access to historical documents—which is and has been RRCHNM’s mission statement since Roy Rosenzweig himself.