Library Guides: Text mining & text analysis: Open sources

Open sources of text data

Australian resources

Australian Data Archive
The Australian Data Archive (ADA) provides a national service for the collection and preservation of digital research data and to make these data available for secondary analysis by academic researchers and other users. Download of data requires approval.
Trove, National Library of Australia
Trove brings together content from Australian libraries, museums, archives and other research organisations. Go to the Advanced Search and type in your search terms. Limit your results to ‘Australian content’ and the format of ‘Thesis’. On the results page, you can refine the availability to ‘Online’ items.
GLAM Workbench
Here you’ll find a collection of tools, tutorials, examples, and hacks to help you work with data from galleries, libraries, archives, and museums (the GLAM sector). The primary focus is Australia and New Zealand, but new collections are being added all the time.

Trove is not just a website for accessing Australian content from libraries, museums, and archives, it's also a platform for building your own tools and resources.

Using the API provides an introduction to Trove's Application Programming Interface (API). If you're a researcher or developer thinking about what you might be able to do with Trove's rich stores of data, look here for support and inspiration.

For example:

Trove Newspaper Harvester
Download large quantities of digitised newspaper articles from Trove's digitised newspaper collection.

General resources

CORE (COnnecting REpositories) - API
CORE indexes research papers from data providers including institutional and subject repositories, preprint servers, and open access and hybrid journals. CORE currently contains 295M open access articles collected from 11K data providers around the world.
Crossref text and data mining
Crossref can be used by researchers to easily harvest full text documents from participating publishers regardless of their business model (e.g. open access, subscription). Provides step-by-step instructions.
Digital Public Library of America
DPLA offers a single point of access to millions of items from libraries, archives, and museums around the United States. Data is available for bulk download in JSON files.
HathiTrust Digital Library
Its corpus is available for research purposes. Learn more at HathiTrust Research Center page. UQ Library contributes library content for digitization and inclusion in this resource. Use UQ login to access full collection.
Google Books
Search full text of books in many languages. Download books in the public domain. The Advanced Search allows you to filter for "full-view".
Google Books BYU View
Compares The Corpus of Historical American English (COHA), Google Books (Standard), and the Google Books (BYU / Advanced) corpus in NGrams.
Google Books Ngram Viewer
Charts the frequencies of any word or short sentence using yearly count of n-grams found in the sources printed between 1500- present. The Ngram Viewer is optimized for quick inquiries into the usage of small sets of phrases. If you're interested in performing a large scale analysis on the underlying data, download of the corpora is available. Texts are in American English, British English, French, German, Italian, Spanish, Russian, Hebrew, and Chinese.
Internet Archive & Open Library
Offers over 10,000,000 fully accessible books and texts. Includes texts, audio, moving images, and software as well as archived web pages in their collection. Instructions for downloading in bulk.
Online Books Page
Lists over 2 million free books on the internet (includes Project Gutenberg, Hathi Trust, Google Books, publisher and institutional archives, etc). Provides a section on non-English language texts.
Project Gutenberg
The first producer of free electronic books (ebooks), their catalog includes over 53,000 free books and over 100,000 titles. Here is the Project's Terms of Use.
Wikidata
Structured data from Wikipedia and other open knowledge bases, available via direct download or API. Wikidata: data access.

Humanities and Social Sciences resources

Chronicling America: Historical American Newspapers
Collection of digitized historical newspapers from 1789-1924. OCR batch downloads available.
Cultoromics Bookworm Viewer
Interface tool for queries in the Google Books corpus. Developed by the Culturomics folks at Harvard it limits itself to only those digitized texts which have information about them (Full title, Publication Date, Publication Place, etc.) on OpenLibrary.org. Users can run queries in highly selective corpora based on subject (books on world history, American books on science, etc.) though these corpora are much smaller than those in the full Google Books collection.
Early English Books Online
EEBO-TCP is a partnership with ProQuest and with more than 150 libraries to generate highly accurate, fully-searchable, SGML/XML-encoded texts corresponding to books from the Early English Books Online Database.
Europeana APIs
Europeana is a digital library with millions of books, paintings, films, museum objects and archival records that have been digitized by more than 2,000 institutions across Europe.
HuNI - Humanities Networked Infrastructure
HuNI (Humanities Networked Infrastructure) combines data from many Australian cultural websites into the biggest humanities and creative arts database (not all full text) ever assembled in Australia. HuNI data covers all disciplines and brings together information about the people, works, events, organisations and places that make up the country's rich cultural landscape.
University of Oxford Text Archive
A repository of digital literary and linguistic resources for research and teaching in higher education.
WordHoard
Contains the entire canon of Early Greek epic in the original and in translation, as well as Chaucer, Shakespeare, and Spenser. Texts are annotated or tagged by morphological, lexical, prosodic, and narratological criteria. User interface allows non-technical users to explore the greatly increased query potential of textual data for computer-assisted study.

Health and Science resources

arXiv.org
arXiv.org is a repository of electronic preprints of scientific papers in the fields of mathematics, physics, astronomy, computer science, quantitative biology, statistics, and quantitative finance, which can be accessed online. Produced by Cornell University.
BioMed Central
Over 315,000 full-text, peer-reviewed science, technology and medicine articles are available for text and data mining.
PLOS
Public Library of Science. Provides access to its peer-reviewed articles. Provides a specific Text Mining Collection.
PubMed Article Datasets
PMC and the NCBI Bookshelf offer several large datasets of journal articles and other scientific publications made available for retrieval under license terms, including PMC Open Access Subset. This subset contains millions of full-text open access article files made available under a Creative Commons or similar license terms or with publisher permission.
PubMed Central Databases and Text Mining Tools
Multiple text mining tools to analyze not only scholarly publications, but also other types of biomedical resources, such as Electronic Health Records.
ScienceDirect TDM
Elsevier have adopted a license-based approach that automatically enables researchers at subscribing institutions to text mine for non-commercial research purposes via an API key, and to gain access to full-text content in XML.

In addition to supporting researchers at subscribing institutions to text mine, our policy also allows everyone to download open access content to mine via our Full Text API, in line with the individual article's user license.
The General Index
Public Resource, a registered nonprofit organization based in California, has created a General Index to scientific journals. The General Index consists of a listing of n-grams, from unigrams to five-grams, extracted from 107 million journal articles.

Law resources

CaseLaw Access Project
CAP includes all official, book-published United States case law — every volume designated as an official report of decisions by a court within the United States.
All state courts, federal courts, and territorial courts. Our earliest case is from 1658, and our most recent cases are from 2018.