A Language Model from 1913: Pretraining on Historical Text

arXiv:2606.02991 · cs.CL, cs.AI · Submitted 2026-06-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "A Language Model from 1913: Pretraining on Historical Text".

Tom: We introduce TYPEWRITERLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we're diving into this paper today, "A Language Model from one thousand nine hundred thirteen: Pretraining on Historical Text," which is seriously interesting because it tackles the whole problem of how to train AI models that actually know things from a specific time period <ref:2606.02991#pg0>. We've got Jane here to help us make sense of all this technical stuff.

Jane: Thanks, Tom. It’s a really focused piece of research because it zeroes in on creating language models that are strictly limited to text written before one thousand nine hundred thirteen which is a huge challenge when dealing with the vastness of modern internet data <ref:2606.02991#pg0>. We need to understand how they managed to build something this specialized and what that means for the future.

Lu: From an AI perspective, what really strikes me about this approach is how they tackle the data quality issue head-on by building this specific historical corpus, TYPEWRITERCORPUS, which spans one thousand seven hundred to one thousand nine hundred thirteen <ref:2606.02991#pg1>. That level of curation sounds incredibly demanding and necessary for achieving any form of reliable historical modeling.

Meng: I'm curious about the practical side here; how did they actually manage the sheer volume of data from institutional books while keeping the noise down? We need to know if this is something that can be scaled up reliably in a real-world engineering environment without it becoming unmanageable or too slow to process.

Lalam: I’m excited about the concept because if we can build models with such strict temporal boundaries, imagine what that means for preserving and understanding historical language accurately in applications like archival research or educational tools. It feels like a step toward creating AI that has a deep, verifiable context rather than just broad knowledge.

Tom: Exactly, Lalam. That strict boundary is the core of the innovation here; they aren't just throwing old books at a model and hoping for the best—they are actively mitigating leakage at every single stage of development to ensure temporal consistency. Jane, can you explain what that mitigation actually involves?

Jane: Certainly. They detail several ways they filter out unwanted data during corpus construction, such as removing OCR artifacts like spaces at visual column boundaries or discarding text fragments that are too heavy on symbols. Then, they have a multi-stage process to remove metadata from institutional books and any web or HTML remnants introduced during processing.

Lu: That systematic filtering sounds incredibly rigorous; it shows they didn't just rely on luck with their data collection but built a robust pipeline for quality control, which is something we always strive for in complex AI systems. It makes sense that the 54Btoken corpus comes from such extensive cleaning procedures <ref:2606.02991#pg0>.

Meng: From an engineering standpoint, those filtering steps sound like they add significant upfront computational overhead during the pre-training phase; we need to make sure that this cleaning doesn't lead to an impossibly long training time or a model that simply forgets how to process normal language structure because too much has been stripped away.

Title and authors: Lalam: I think the systematic nature of it is what’s important; it shows there’s a repeatable methodology for ensuring data integrity across different historical sources, which is something we can definitely apply to other specialized domains where data provenance matters immensely.

Tom: It really does; and that leads us directly into the next big part of their work: how they fine-tune the model for instruction following while keeping it grounded in those old texts. They introduce lexically grounded instruction tuning, which is a clever way to enforce that temporal constraint during post-training.

Jane: That method involves a strict rule: any response the model gives must be directly derivable from the historical source documents or a very small allowlist of function words. It’s an incredibly tight leash on the model's output, ensuring it stays rooted in one thousand nine hundred thirteen English <ref:2606.02991#pg0>.

Lu: The creation of HISTORYLIMA and HISTORYSELFINSTRUCT datasets illustrates how they’ve built specific training sets designed to enforce this grounding; especially the self-instruct method where only the instructions are generated while responses stay fixed historical anchors is a really creative way to guide the learning process.

Meng: I see how that works, but ensuring that "derivable from" rule is robust across different types of queries or complex reasoning tasks could be a significant hurdle when we move this to more general use cases. We need to know if it breaks under pressure.

Lalam: But the goal here isn't general capability; it's historical fidelity, and that constraint is what makes the output trustworthy for specific research needs where accuracy about the period is paramount. It’s about creating a reliable tool for understanding historical discourse.

Tom: And to test this consistency, they didn't just stop at training; they built HISTORYEVENT, which is a benchmark of two thousand three hundred forty-four significant historical events spanning from one thousand seven hundred all the way up to two thousand twenty-five <ref:2606.02991#pg1,a benchmark of 2,344 significant historical events spanning>. This lets them measure if the model actually learns the temporal limits correctly.

Jane: They used perplexity-based surprisingness evaluation on that set, and they found that History LMs become substantially more surprised by events after their cutoff compared to a modern baseline model like Llama-three point one-8B, which stays relatively flat across time <ref:2606.02991#pg0>. That suggests the one thousand nine hundred thirteen cutoff is actively reflected in how the model processes new information <ref:2606.02991#pg0>.

Lu: That finding is really telling; it shows that simply training on old data doesn't automatically mean the model won't react to newer context, but here, the effect of that reaction becomes measurable and predictable based on what we expect from a historical boundary. It connects knowledge cutoff directly to model behavior in a way I hadn't fully appreciated.

Meng: So they are showing that the temporal constraint isn't just an abstract idea; it has a tangible effect on the perplexity scores when encountering events outside the training window, which gives us some concrete data for how to measure that fidelity.

Title and authors: Lalam: That predictability is exactly what makes this research valuable for safety and verification; if we can measure *how much* a model reacts to out-of-scope information, it helps us build guardrails for applications where temporal accuracy matters. It’s about knowing the limits of the knowledge being used.

Tom: To wrap up on these improvements, the authors suggest looking into studying the interplay between reasoning and memorization in these models, and also investigating how scaling affects their behavior when they are already data-limited, which points toward future research directions for this work.

Jane: The primary limitation they admit is that they don't systematically study the effect of changing dataset composition or mixing ratios during pre-training, meaning we don't have a complete picture of how varying the input data density impacts performance. They also note that training these models is inherently data-constrained, which means exploring more data-efficient strategies for building them is still an open area.

Lu: That limitation is important; understanding the reasoning versus memorization trade-off will be key to figuring out if this type of constraint helps or hinders the model’s ability to actually reason about historical context rather than just recalling specific phrases from the training set.

Meng: From an engineering view, that lack of systematic study on data composition means we can't easily design a scalable pipeline that automatically optimizes for the best data mix without extensive manual tuning, which complicates deployment planning.

Lalam: But even with these limitations, this work establishes a strong foundation for developing models where temporal fidelity is not an afterthought but a core requirement, which opens doors for applications that demand high levels of historical accuracy. This paper sets a clear path forward for building trustworthy AI in specialized fields.

Tom: It certainly does establish that the TYPEWRITERLM framework is a viable method for creating competitive, historically grounded language models by focusing on rigorous data mitigation and instruction tuning techniques, all while providing measurable metrics for temporal consistency.

Jane: We've seen how they constructed the 54Btoken corpus and how they used lexically grounded instruction tuning to constrain the model's responses tightly to the pre-one thousand nine hundred thirteen text <ref:2606.02991#pg0>. It’s a very detailed technical process that shows how precision in data handling leads directly to precision in historical output.

Lu: And their evaluation suite, HISTORYEVENT, which uses perplexity-based surprisingness evaluation against events from one thousand seven hundred to two thousand twenty-five provides a concrete way to see if the intended knowledge cutoff is actually being respected by the model's learned distributions <ref:2606.02991#pg1>.

Meng: So, in summary, this paper demonstrates that with careful construction of the corpus and a strict post-training constraint on responses, we can produce language models that perform well on general tasks while exhibiting measurable temporal characteristics consistent with their training data.

Lalam: It really shows that focusing intensely on data provenance and enforcing lexical grounding during instruction tuning is a powerful way to build AI systems where historical accuracy is a non-negotiable feature. This work on the TYPEWRITERLM paper provides a solid blueprint for anyone interested in building tools for deep historical analysis.

The paper's summary: Tom: So, to recap this whole paper, they've built TYPEWRITERLM, which is a seven point 24B parameter language model that was trained exclusively on English text from before one thousand nine hundred thirteen to solve data quality issues and temporal leakage in historical AI research.

Jane: That’s right; the core idea is creating a model strictly grounded in its era, using a highly curated corpus of about fifty-four billion tokens derived mostly from institutional books.

Lu: What I find really fascinating is how they manage that data cleaning process—removing OCR errors and metadata like ownership stamps—it shows a deep understanding of how to handle messy historical archives for AI.

Meng: From an engineering standpoint, the strict filtering pipeline they use sounds incredibly robust; we need to figure out if that level of noise reduction can be standardized across different historical domains without needing custom cleaning scripts for every new corpus.

Lalam: And from a cultural perspective, this work is significant because it allows us to create AI tools that preserve and analyze the language and thought patterns of the past with a verifiable temporal boundary, which could reshape how we study historical discourse.

Tom: Exactly; it's about building something that doesn't drift into modern biases or knowledge, giving us a model we can actually trust when looking at old documents.

Jane: The researchers also showed that even with these constraints, the model still shows patterns of increasing surprise when it encounters information from after one thousand nine hundred thirteen in their evaluation suite.

Lu: That surprisingness metric is really telling; it proves that the knowledge cutoff isn't just a hard stop, but something that visibly affects how the AI processes new data relative to its training history.

Meng: So they have a measurable way to quantify temporal consistency, which moves this from a conceptual idea into something we can test and debug with actual numbers.

Lalam: This is huge because it gives us a concrete way to measure fidelity; if we can reliably track that surprise metric, we can build systems that are auditable for historical accuracy.

Tom: It really does give us confidence in the output, showing that data constraints can actually lead to competitive reasoning capabilities on general benchmarks like ARC-Challenge.

Jane: And it’s not just about old books; they showed that by fine-tuning with lexically grounded instructions, you can force the model to stay tightly locked into the historical vocabulary during its responses.

Lu: That self-instruct method for instruction tuning is really clever because it inverts the process, letting them generate instructions based on fixed historical anchors, which helps ensure those linguistic constraints stick.

Meng: That instruction tuning sounds like a great way to prevent the model from hallucinating modern phrasing when it’s supposed to be speaking in an older style.

Lalam: The implication for culture is that we could have AI assistants that converse or analyze historical texts with a genuine sense of period authenticity, which enriches our understanding of past societies immensely.

Tom: Absolutely; think about the impact on education and research where having a strictly temporal model means you’re getting an uncompromised view of how ideas were expressed at a certain time.

Jane: So while they acknowledge limitations, specifically that they don't systematically study all possible data mixing ratios during pre-training, the work still shows a clear path for creating these temporally constrained models.

Lu: That lack of systematic study on data composition is definitely an avenue for future research; exploring how different mixes of historical and modern text affect reasoning might unlock new ways to guide this type of model development.

Meng: For practical deployment, knowing the exact limitations on dataset composition means we can design better pipelines that manage that risk proactively instead of just hoping the training data is perfect.

Lalam: Ultimately, this research paves the way for building AI systems where historical context isn't just a feature but a fundamental guarantee of accuracy in applications that deal with human history.

The paper's improvements: Tom: So, we've seen that TYPEWRITERLM is built on a very strict foundation of historical data and intense filtering to handle temporal leakage, and now they’re looking ahead at how to make it even better.

Jane: That’s right; the authors aren't just happy with the initial results; they’re pointing toward specific next steps for refinement, which shows how much work is still needed to push this kind of model further.

Lu: They suggest studying the relationship between reasoning and memorization in these models because understanding that interplay will help them decide if they should focus more on making it reason historically or just memorize old sentences.

Meng: That makes sense; knowing where the model sits on that spectrum is critical for determining whether we need to adjust the training process toward more complex historical inference or tighter factual recall.

Lalam: And I think that’s incredibly important because if we can map out that reasoning-memorization balance, it could help us design AI assistants with much more nuanced and trustworthy historical perspectives on a wide range of topics.

Tom: They also noted that they don't have a systematic study on how changing the mix of different datasets during pre-training affects performance, which means exploring those compositional strategies is a definite future direction.

Jane: That’s fair; even with such careful pre-training, not systematically testing different data combinations means we haven't fully mapped out the optimal recipe for building these historical models.

Lu: I think that suggests future work could involve designing automated methods to tune the input data composition based on what we observe about their reasoning performance.

Meng: From an engineering side, if they can figure out how to optimize that data mix, it would allow us to build a more flexible and robust system that isn't entirely dependent on one specific set of pre-training data.

Lalam: That flexibility is vital for cultural impact; imagine having an AI capable of adapting its historical focus based on the context of the user’s question without losing its core temporal integrity.

Tom: And they also flagged that as these models scale up, we need to be more mindful because they anticipate potential societal risks associated with these highly specialized, temporally constrained language models.

Jane: That's a very responsible caution; it highlights that even highly specialized tools require careful consideration regarding their broader use and safety implications in the world.

Lu: That thinking about scaling behavior under data-limited settings is fascinating; it opens up questions about how these models behave when they are forced to operate with restricted knowledge.

Meng: We need to keep that in mind during development; anticipating those scaling behaviors now will prevent unexpected issues down the line when we move toward larger parameter counts.

Lalam: The future vision here is creating AI assistants that can be deeply rooted in a specific historical time period, offering insights into past languages and social structures with unparalleled fidelity, which could profoundly change how we approach human history.

Conclusion: Tom: So, to wrap up this deep dive into "A Language Model from one thousand nine hundred thirteen: Pretraining on Historical Text," we’ve seen how researchers can build models that are extremely precise about their knowledge cutoff through rigorous data cleaning and instruction tuning methods.

Jane: It really shows that you don't need massive, modern datasets to create an AI with deep, verifiable context if you focus on making the pre-training corpus as clean and temporally constrained as possible.

Lu: The potential here is wild; imagine having specialized models for every historical discipline—literature, science, social history—all operating within their own perfect temporal bubble.

Meng: From an engineering standpoint, this methodology gives us a clear blueprint for building systems where temporal fidelity is a primary design constraint rather than something that has to be patched on later.

Lalam: The cultural implication is huge because it means we can create AI tools that interact with the past in a way that feels authentic, which could significantly deepen our connection to historical knowledge across society.

Tom: Exactly; this paper proves that data-constrained pre-training can result in language models that are competitive and reliable for tasks requiring strict temporal accuracy.

Jane: And we saw how they use perplexity evaluation on the HISTORYEVENT benchmark to actually measure if the model is learning those temporal boundaries correctly, which is a really strong validation step.

Lu: It confirms that the learned knowledge distribution isn't just a guess; it’s measurable, which opens up avenues for more sophisticated temporal modeling in general AI architectures.

Meng: For deployment, knowing how to rigorously constrain the model’s output during instruction tuning provides a practical safety layer that helps mitigate some of those risks we discussed earlier.

Lalam: It gives us a framework for developing AI assistants that can serve as true historical interpreters, providing nuanced and accurate information about specific eras without mixing in modern biases.

Tom: So, to sum up, the work on "A Language Model from one thousand nine hundred thirteen: Pretraining on Historical Text" shows a very successful path toward creating highly specialized language models for historical analysis through meticulous corpus construction and constrained instruction tuning.

Jane: It’s a powerful demonstration of how precision in data handling directly translates into precision in the model's ability to handle temporal information, which is something we all need to pay attention to.

Lu: The next big thing we should look at is how this kind of temporal grounding could integrate with other modalities, perhaps combining historical text with visual or audio data for richer historical simulations.

Meng: That’s a valid direction; integrating these temporally constrained language models into multimodal systems could create truly context-aware digital archives.

Lalam: I’m really looking forward to seeing how this research informs the next generation of AI tools that can help us understand the past in such a structured and reliable way.

Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber, Yixuan Wang, Junchi Yu, Freda Shi, Philip Torr, *Yao Lu

University of Waterloo · Vector Institute

cs.CL, cs.AI

Submitted: 2026-06-02

Updated: 2026-10-02

Comments: Accepted by EMNLP 2026

Code: https://github.com/openai/tiktoken

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: We introduce TYPEWRITERLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913, addressing challenges in data quality and temporal leakage to create historically

Key concepts

TYPEWRITERLM: A 7.24B-parameter decoderonly Transformer model trained only on English text predating 1913. It was built with strict knowledge cutoff and specific data cleaning methods to ensure it remains historically accurate for language tasks.
Leakage Mitigation Strategies: A multi-stage process used during training to prevent the model from learning information outside its historical cutoff. This involved filtering metadata from books, removing web artifacts, and using lexically grounded instruction tuning to force responses to stick strictly to historical source text.
Temporal Consistency: The ability of a language model trained on a specific time period (like pre-1913) to correctly reflect that knowledge. Evaluation showed that History LMs become more surprised by events occurring after their cutoff, confirming the intended temporal constraints are reflected in their knowledge.

Terminology

Benchmark — 29× in this paper · explained in 2 episodes
Instruction Tuning — 27× in this paper · explained in 2 episodes
Accuracy — 11× in this paper · explained in 1 episode
Consistency — 9× in this paper · explained in 3 episodes
Grounding — 9× in this paper · explained in 3 episodes
Fidelity — 8× in this paper · explained in 2 episodes
Temporal consistency — 8× in this paper · explained in 2 episodes
Transformer — 6× in this paper · explained in 2 episodes
Core — 5× in this paper · explained in 2 episodes
Perplexity — 5× in this paper · explained in 3 episodes
Provenance — 5× in this paper · explained in 4 episodes
Fine-Tuning — 3× in this paper · explained in 5 episodes
Adaptation — 2× in this paper · explained in 1 episode
Corpus of English Dialogues — 2× in this paper
Data Provenance — 2× in this paper · explained in 2 episodes
Distribution Shift — 2× in this paper · explained in 6 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Verification — 2× in this paper · explained in 2 episodes
Adam — 1× in this paper · explained in 2 episodes
Alignment — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 4 episodes
Commentary — 1× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Domain Adaptation — 1× in this paper · explained in 5 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
LoRA Fine-Tuning — 1× in this paper · explained in 3 episodes
Positional Bias — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Synthetic Data Generation — 1× in this paper · explained in 5 episodes
Tokenization — 1× in this paper · explained in 2 episodes
Transformer Models — 1× in this paper · explained in 2 episodes
Verifier — 1× in this paper · explained in 2 episodes

Summary

We introduce TYPEWRITERLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913, addressing challenges in data quality and temporal leakage to create historically grounded models for NLP research.

Core Model and Corpus Construction

The paper introduces TYPEWRITERLM, a 7.24B-parameter decoderonly Transformer following the Llama 3 architecture, which is trained with a strict knowledge cutoff of 1913. To address data quality and availability issues, the authors constructed TYPEWRITERCORPUS, a 54Btoken historical corpus collected from diverse archival and linguistically annotated sources with extensive data cleaning and leakage mitigation procedures. This corpus spans 1700–1913, primarily derived from Institutional Books (Cargnelutti et al., 2025), which constitutes 97.7% of the total corpus. The construction process involved applying several normalization and filtering procedures to remove OCR artifacts, such as spaces at visual column boundaries, and discarding symbol-heavy fragments.

Leakage Mitigation Strategies

The authors detail a multi-stage approach to control leakage at every pipeline step:

They apply rule-based filtering to mitigate Institutional provenance metadata, removing elements like ownership stamps or catalog identifiers from institutional books.
Web and HTML artifacts introduced during corpus processing are removed from all datasets.
Publishing and editorial metadata, such as title-page imprints, is removed heuristically.
For post-training, they introduce lexically grounded instruction tuning, where responses are constrained to remain directly grounded in historical source documents. This involves a strict constraint: a response is accepted only if every lexical token is derivable from the source text or a small allowlist of function words.

Instruction Tuning Datasets

To implement lexically grounded instruction tuning, two datasets were constructed:

HISTORY-LIMA: A small high-quality instruction-tuning set consisting of 1,000 lexically grounded single-turn examples. This dataset includes a multi-turn variant with 30 dialogues and 15 dialogues curated from pre-1913 dialogue texts, such as catechisms and literary exchanges.
HISTORYSELFINSTRUCT: Inspired by SelfInstruct, this dataset scales instruction tuning by inverting the pipeline: responses remain fixed historical anchors, while only instructions are model-generated. This process involves three stages: (i) Seed construction using HISTORY-LIMA examples, (ii) Question-generator training using LoRA to obtain a generator that produces historically consistent instructions conditioned on grounded responses, and (iii) Filtered generation at scale.

Evaluation Framework

Evaluation assesses both downstream capability and temporal consistency using two main components:

Downstream Capability: The model is tested on general benchmarks like Hellaswag, ARC-Easy, and ARC-Challenge. Results show that TYPEWRITERLM achieves performance comparable with other History LMs trained using modern LLM supervision, demonstrating nontrivial reasoning capability.
Temporal Consistency: They construct HISTORYEVENT, a benchmark suite of 2,344 significant historical events spanning 1700–2025. They use perplexity-based surprisingness evaluation to find that History LMs become substantially more surprised by post-cutoff events, suggesting the intended historical cutoffs are meaningfully reflected in the models’ learned knowledge distributions.

Key Findings and Limitations

The research demonstrates that data-constrained pretraining can produce competitive, historically grounded language models. The evaluation reveals that while History LMs show patterns of rising surprisingness post-cutoff, they still suffer from lookahead bias, as they find leakage for the two largest models on the HISTORY-EVENT dataset. Furthermore, performance on modern benchmarks is influenced by temporal mismatch in language style; rewriting examples into pre-1800 context yields substantially larger gains for History LMs compared to modern LLMs. The work concludes that developing leakage-free History LMs is important for applications requiring strict temporal fidelity.

Future Directions and Limitations

The paper suggests future research directions include studying the reasoning–memorization interplay and investigating how scaling behavior manifests under data-limited pretraining settings. A primary limitation acknowledged is that they do not systematically study the effect of dataset composition or mixing ratios during pre-training, and training History LMs is inherently data-constrained, leaving exploration of data-efficient strategies to future work. Additionally, the authors note that as these models scale, they may pose societal risks, necessitating future safety warnings and guardrails.

**(Note: The summary adheres strictly to the constraints—no headers other than the required bold section headers, no meta-commentary, and focuses only on information present in the text.

Improvements for AI systems

As a fastidious researcher, I have analyzed the TYPEWRITERLM framework and its associated pipeline for improving AI systems, particularly in the domain of historical language modeling.

Here are the specific improvements that can be implemented based on this paper:

) 1. Implement a Strict Temporal Knowledge Cutoff Enforcement Layer

The system must utilize a hard knowledge cutoff (e.g., 1913 for TYPEWRITERLM) enforced not just during pre-training but also through post-training constraints. This prevents lookahead bias by explicitly excluding any information beyond the cutoff date from the model's learned parameters.

Develop a Multi-Stage Leakage Mitigation Pipeline

Integrate rigorous data cleaning procedures into the pre-training corpus construction (TYPEWRITERCORPUS) to systematically remove temporal leakage sources:

Remove institutional provenance metadata (e.g., library stamps, catalog identifiers).
Filter out web and HTML artifacts (URLs/entities).
Heuristically strip publishing and editorial metadata from all text sources.

Implement Lexically Grounded Instruction Tuning (LGI) for Post-Training

Replace standard instruction tuning methods that rely on modern data or frontier LLM imitation with the LGI framework:

Constrain all model responses during fine-tuning to be derivable exclusively from the pre-1913 source documents and a small allowlist of function words.
Employ a strict post-generation verifier that checks for exact lexical matching against the source text, allowing only limited, contextually valid morphological normalization or page reconstruction corrections.

Construct a Leakage-Aware Evaluation Suite (HISTORY-EVENT)

Develop benchmarks designed to test both downstream capability and temporal consistency:

Use event descriptions from historical timelines (1700–2025).
Employ two metrics: BPB Surprisingness (to measure how surprised the model is by post-cutoff events) and Recall/Leakage testing (Strict vs. Relaxed criteria to quantify actual data leakage of post-cutoff information).

Create Specialized Instruction Datasets for Fine-Tuning

Generate instruction tuning data specifically designed for historical fidelity:

Construct HISTORY-LIMA: A small set of high-quality, lexically grounded single-turn examples (1,000 pairs) to minimize drift from the pretraining distribution.
Construct HISTORYSELFINSTRUCT: Scale instruction tuning by inverting the pipeline—generate only instructions based on fixed historical anchors—to train a generator that produces historically consistent prompts without introducing modern linguistic priors.

The improved AI system (TYPEWRITERLM) can perform the following specific tasks:

Perform high-fidelity historical text generation: The model will generate responses that are guaranteed to be lexically grounded in pre-1913 documents, ensuring stylistic and vocabulary consistency with the target era, thus avoiding anachronistic language or modern slang.
Demonstrate temporally consistent historical reasoning: When asked about events within its knowledge cutoff (pre-1913), it will demonstrate strong reasoning capabilities comparable to other models (e.g., achieving competitive scores on ARC-Challenge) while exhibiting a measurable, predictable increase in surprisingness when presented with post-cutoff events.
Serve as a reliable tool for historical social science research: By rigorously controlling leakage and demonstrating verifiable temporal boundaries, the system can be trusted for tasks requiring strict fidelity to historical knowledge constraints (e.g., analyzing 19th-century parliamentary discourse or Victorian literature).

Abstract

While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text) in a data-constrained setting can produce a temporally grounded language model that still shows reasonable performance on language understanding. However, developing History LMs requires addressing challenges in data quality, preventing temporal leakage in post-training, and constructing temporally aligned evaluations. We address these challenges and pretrain TypewriterLM, a 7.24B-parameter model with a 1913 knowledge cutoff. We construct TypewriterCorpus, a 54B-token historical corpus with extensive temporal filtering, propose lexically grounded instruction tuning that constrains all responses to vocabulary from historical source documents, and introduce History-Event, a benchmark of 2,344 events for evaluating both competence and cutoff adherence. We release TypewriterLM and all associated resources to support future research on History LMs.