Skip to Main Content

Text and Data Mining

A How Do I Guide that covers resources for Text and Data Mining through the University Library

Working with TDM

There is no one way to go about conducting a text and data mining project, but it is important to account for deadlines and other time constraints when planning your approach. Resources may be unavailable for use, or there may be complications in accessing them for TDM purposes, so it worthwhile to arrange alternative options. The following method is an example of the steps to take in a TDM project, but is entirely subject to change based on your own needs, circumstances, and skills.

  1. Evaluate your research needs. How will TDM help you to achieve these? What are your goals and expectations? Establishing these now will be helpful when it comes to selecting resources.
  2. Locate the necessary resources to be used for TDM, ensuring that your selections align with ethical and legal considerations. Remember that you may need to acquire permission to use specific resources, and you should never assume that any text or data is inherently okay to use for TDM. It is also good practice to inform relevant publishers of any intent to perform TDM using their resources. Make sure that you have allowed sufficient time for this step.
  3. Assess which tools will work best for your project. How unique are your research needs, and what software or programs will help you to meet them? Do you have coding or programming skills? Is there any kind of assistance that you might find helpful?
  4. Clean and prepare text or data. Resources may not immediately be ready for use with TDM tools, and cleaning and preparation may be necessary. This involves processes such as stop word removal and punctuation removal in order to standardise text and data.
  5. Perform TDM. Make sure to follow any rules or guidelines specified by publishers, and to keep a record of all resources and tools used in the TDM process.
  6. Acknowledge and credit use of TDM resources and tools. Once your research is complete, ensure that you have appropriately and correctly acknowledged, credited, and cited any and all resources and tools used throughout the research project, including the TDM process.