In 2023, a group of researchers who studied dishonesty and unethical behavior became big news for all the wrong reasons. According to an article by the Chronicle of Higher Education,^[1] the findings of the article “Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End,” by Lisa L. Shu, Nina Mazar, Francesca Gino, and Max H. Bazerman,^[2] were unable to be replicated in later studies. This caught the attention of scholars who then analyzed the data from the original and the later studies. These scholars found evidence that data from the original study had been fabricated,^[3] which led to the article’s retraction.

How did an article with fabricated data end up published?  Was the editor at fault for not scrutinizing the article closely enough?  Did the peer reviewers do a shoddy job of checking the data analysis?  Were the authors at fault for falsifying the data, or were the authors provided incorrect or falsified data by others?

There are a number of issues at play in this case, but there’s no question that evaluating sources is an important part of the research process and doesn’t just apply to internet sources. Using inaccurate, irrelevant, or poorly researched sources can affect the quality of your own work. Being able to understand and apply the concepts that follow is crucial to becoming a more savvy user and creator of information.

A reader of “Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End”  may not have reason to believe the information the author reports from their research is inaccurate, but if they compared the article against other research studying the same topic, the inconsistencies might become more apparent.

People who are adept at evaluating information are able to review the research process and compare and judge information and data. Individuals who evaluate information well:

Understand:

• The information and data landscape of their learning/research context;
• Issues of quality, accuracy, relevance, bias, reputation and credibility relating to information and data sources;
• How information is evaluated and published to help inform their personal evaluation process;
• The importance of consistency in data collection;
• The importance of citation in their learning/research context.

Are able to:

• Distinguish between different information resources and the information they provide;
• Choose suitable material on their search topic using appropriate criteria;
• Assess the quality, accuracy, relevance, bias, reputation and credibility of the information resources found;
• Assess the credibility of the data gathered;
• Read critically, identifying key points and arguments;
• Relate the information found to the original search strategy;
• Critically appraise and evaluate their own findings and those of others;
• Know when to stop.

Distinguishing Between Information Resources

Information is published in a variety of formats, each with its own special considerations when it comes to evaluation. When conducting your research, consider the following formats and the different ways in which each format is developed and presented.

Social Media

Social media is a quickly rising star in the landscape of information gathering. Facebook updates, tweets, wikis, and blogs have made information creators of us all and have a strong influence not just on how we communicate with each other but also on how we learn about current events or discover items of interest. Anyone can create or contribute to social media and nothing that’s said is checked for accuracy before it’s posted for the world to see. So do people really use social media for research? Currently, the main use for social media, like tweets and Facebook posts, is as primary sources that are treated as the objects under study rather than sources of information on a topic. But now that the Modern Language Association (MLA) has a recommended way to cite a tweet, social media may, in fact, be gaining credibility as a resource.

Websites

In addition to social media, a common place to find information is websites. Commercial organizations, educational institutions, government entities, research think tanks, and more publish blogs, reports, and statistics that could be relevant to your research. However, evaluating the credibility of web sources can be challenging. Students are often taught in high school, as a general rule of thumb, to avoid .com websites as untrustworthy and to use .org and .edu domains instead. In research, it is important to critically evaluate all web sources for trustworthiness, accuracy, and currency, regardless of domain suffix. Websites are considered popular sources, even if published by an educational institution or a government entity. Some websites, including some government websites, may be considered primary sources, which could have value in research.

News Articles

These days, social media will generally be among the first to cover a big news story, with news media writing an article or report after more information has been gathered. News articles are written by journalists who either report on an event they have witnessed firsthand, or after making contact with those more directly involved. The focus is on information that is of immediate interest to the public and these articles are written in a way that a general audience will be able to understand. These articles go through a fact-checking process, but when a story is big and the goal is to inform readers of urgent or timely information, inaccuracies may occur. In research, news articles are often best treated as primary sources, especially if they were published immediately after a current event.

Magazine Articles

While news articles and social media tend to concentrate on what happened, how it happened, who it happened to, and where it happened, magazine articles are more about understanding why something happened, usually with the benefit of at least a little hindsight. Writers of magazine articles also fall into the journalist category and rely heavily on investigation and interviews for research. Fact-checking in magazine articles tends to be more accurate because magazines publish less frequently than news outlets and have more time to get the facts right. Depending on the focus of the magazine, articles may cover current events or just items of general interest to the intended audience. The language may be more emotional or dramatic than the factual tone of news articles, but the articles are written at a similar reading level so as to appeal to the widest audience possible. A magazine article is considered a popular source rather than a scholarly one which gives it less weight in a research context but doesn’t take away the value completely.

Scholarly Articles

Scholarly articles are written by, and for, experts in a field and generally describe formal research studies or experiments conducted to provide new insight on a topic rather than reporting current events or items of general interest. You may have heard the term “peer review” in relation to scholarly articles. This means that before an article is published, it undergoes a review process in order to confirm that the information is accurate and the research it discusses is valid. This process adds a level of credibility to the article that you would not find in a magazine or news article. Scholarly articles tend to be long and feature specialized language that is not easily understood by someone who does not already have some level of expertise on the topic. Though they may not be as easy to use, they carry a lot of weight in a research context, especially if you are working in a field related to science or technology. These sources will give you information to build on in your own original research.

Books

Books have been a staple of the research process since Gutenberg invented the printing press because a topic can be covered in more depth in a book than in most other types of sources. Also, the conventional wisdom for books is that anyone can write one, but only the best ones get published. This is becoming less true as books are published in a wider variety of formats and via a wider variety of venues than in previous eras, which is something to be aware of when using a book for research purposes. For now, the editing process for formally published books is still in place and research in the humanities, which includes topics such as literature and history, continues to be published primarily in this format.

Choosing Materials

When choosing a source for your research, what criteria do you usually use? Gauging whether the source relates to your topic at all is probably one. How high up it appears on the results list when you search may be another. Beyond that, you may base your decision at least partly on how easy it is to access.

These are all important criteria, to varying degrees, but there are other criteria you may want to keep in mind when deciding if a source will be useful to your research.

Quality

Scholarly journals and books are traditionally considered to be higher quality information sources because they have gone through a more thorough editing process that ensures the quality of their content. Generally, you also pay more to access these sources or may have to rely on a library or university to pay for access for you. Information on the Internet can also be of a high quality but there is less of a quality assurance process in place for much of that information. In the current climate, the highest quality information, even on the Internet, often requires a subscription or other form of payment for access.

Clues to a source’s level of quality are closely related to thinking about how the source was produced, including what format it was published in and whether it is likely to have gone through a formal editing process prior to publication.

Accuracy

A source is accurate if the information it contains is correct. Sometimes it’s easy to tell when a piece of information is simply wrong, especially if you have some prior knowledge of the subject. But if you’re less familiar with the subject, inaccuracies can be harder to detect, especially when they come in subtler forms such as exaggerations or inconsistencies.

To determine whether a source is accurate, you need to look more deeply at the content of the source including where the information in the source comes from and what evidence the author uses to support their views and conclusions. It also helps to compare your source against another source. A reader of Our Virginia may not have reason to believe the information the author cites from the Sons of Confederate Veterans website is inaccurate, but if they compared the book against another source, the inconsistencies might become more apparent.

Relevance

Relevance has to do with deciding whether the source actually relates to your topic and, if it does, how closely it relates. Some sources may be an exact match; for others, you may need to consider a particular angle or context before you can tell whether the source applies to your topic. When searching for relevant sources, you should keep an open mind—but not too open. Don’t pick something that’s not really related just because it’s on the first page or two of results or because it sounds good.

You can assess the relevance of a source by comparing it against your research topic or research question. Keep in mind that the source may not need to match on all points, but it should match on enough points to be usable for your research beyond simply satisfying a requirement for an assignment.

Bias

An example of bias is when someone expresses a view that is one-sided without much consideration for information that might negate what they believe. Bias is most prevalent in sources that cover controversial issues where the author may attempt to persuade their readers to one side of the issue without giving fair consideration to the other side of things. If the research topic you are using has ever been the cause of heated debate, you will need to be especially watchful for any bias in the sources you find.

Bias can be difficult to detect, particularly when we are looking at persuasive sources with which we want to agree. If you want to believe something is true, chances are you’ll side with your own internal bias without consideration for whether a source exhibits bias. When deciding whether there is bias in a source, look for dramatic language and images, poorly supported evidence against an opposing viewpoint, or a strong leaning in one direction.

Reputation

Is the author of the source you have found a professor at a university or a self-published blogger? If the author is a professor, are they respected in their field or is their work heavily challenged? What about the publication itself? Is it held in high regard or relatively unknown? Digging a little deeper to find out what you can about the reputation of both the author and the publication can go a long way toward deciding whether a source is valuable.

You can investigate the reputation of an author by looking at any biographical information that is available as part of the source. Looking to see what else the author has published and whether this information has positive reviews is also important in establishing whether the author has a good reputation. The reputation of a publication can also be investigated through reviews, word-of-mouth by professionals in the field, or online databases that keep track of statistics related to a journal’s credibility.

Credibility

Credibility has to do with the believability or trustworthiness of a source based on evidence such as information about the author, the reputation of the publication, and how well the source is formatted. How likely would you be to use a source that was written by someone with no expertise on a topic or a source that appeared in a publication that was known for featuring low quality information? What if the source was riddled with spelling and formatting errors? Looking at sources like these should inspire more caution.

Objectively, credibility can be determined by taking into account all of the other criteria discussed for evaluating a source. Knowing that some types of sources, such as scholarly journals, are generally considered more credible than others, such as self-published websites, may also help. Subjectively, deciding whether a source is credible may come down to a gut feeling. If something about a source doesn’t sit well with you, you may decide to pass it over.

Identifying Key Points and Arguments

Evaluating information about the source from its title, author, and summary information is only the first step. The evaluation process continues when you begin to read the source in more detail and make decisions about how (or whether) you will ultimately use it for your own research.

When you begin to look more deeply at your source, pay close attention to the following features of a document.

Introduction

The purpose of the introduction to any piece that has one is to give information about what the reader can expect from the source as a whole. There are different types of introductions including forewords and prefaces that may be written by the author of the book or by someone else with knowledge of the subject. Introductory sections can include background information on why the topic was chosen, background on the author’s interest in the topic, context pertaining to why the topic is important, or the lens through which the topic will be explored. Knowing this information before diving into the body of the work will help you understand the author’s approach to the topic and how it might relate to the approach you are taking in your own research.

Table of Contents

Most of the time, if your source is a book or an entire website, it will be divided into sections that each cover a particular aspect of the overall topic. It may be necessary to read through all of these sections in order to get a “big picture” understanding of the information being discussed, or it may be better to concentrate only on the areas that relate most closely to your own research. Looking over the table of contents or menu will help you decide whether you need the whole source or only pieces of it.

List of References

If the source you’re using is research-based, it should have a list of references that usually appear at the end of the document. Reviewing these references will give you a better idea of the kind of work the author put into their own research. Did they put as much work into evaluating their sources as you are? Can you tell from the citations if the sources used were credible? When were they published? Do they represent a fair balance of perspectives or do they all support a limited point of view? What information does the author use from these sources and in what way does he or she use that information? Use your own research skills to spy on the research habits of others to help you evaluate the source.

Evaluating Your Findings

In the case of “Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End,” the authors used inaccurate or fabricated data as part of their research and this data affected the accuracy of their findings. Likewise, if anyone had used this article as part of their research, it would have set off a chain reaction, since whatever information they cited from “Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End” would naturally have to be called into question, possibly diminishing the value of their own conclusions.

Evaluating the sources you use for quality, accuracy, relevance, bias, and credibility is a good first step in making sure this doesn’t happen, but have you ever thought about evaluating the sources used by your own sources? This takes extra time, but looking at the reference list, bibliography, or notes section of any source you use to gauge the quality of the research done by the author of that source can be an important extra step.

Knowing When to Stop

For some researchers, the process of searching for and evaluating sources is a highly enjoyable, rewarding part of doing research. For others, it’s a necessary evil on the way to constructing their own ideas and sharing their own conclusions. Whichever end of the spectrum you most closely identify with, here are a few ideas about the ever-important skill of knowing when to stop.

1. You’ve satisfied the requirements for the assignment and/or your curiosity on the topic. If you’re doing research as part of a course assignment, chances are you’ve been given a required number of sources. Novice researchers may find this number useful to understand how much research is considered appropriate for a particular topic. However, a common mistake is to focus more on the number of sources than on the quality of those sources. Meeting that magic number is great, but not if the sources used are low quality or otherwise inappropriate for the level of research being done.
2. You have a deadline looming. Nothing better inspires forward motion in a research project than having to meet a deadline, whether it’s set by a professor, an advisor, a publisher, or yourself. Time management skills are especially useful, but since research is a cyclical process that sometimes circles back on itself when you discover new knowledge or change direction, planning things out in minute detail may not work. Leaving yourself enough time to follow the twists and turns of the research and writing process goes a long way toward getting your work in when it’s expected.
3. You need to change your topic. You’ve been searching for information on your topic for a while now. Every search seems to come up empty or full of irrelevant information. You’ve brought your case to a research expert, like a librarian, who has given advice on how to adjust your search or how to find potential sources you may have previously dismissed. Still nothing. It could be that your topic is too specific or that it covers something that’s too new, like a current event that hasn’t made it far enough in the information cycle yet. Whatever the reason, if you’ve exhausted every available avenue and there truly is no information on your topic of interest, this may be a sign that you need to stop what you’re doing and change your topic.
4. You’re getting overwhelmed. The opposite of not finding enough information on your topic is finding too much. You want to collect it all, read through it all, and evaluate it all to make sure you have exactly what you need. But now you’re running out of room on your flash drive, your Dropbox/Google/One Drive account is getting full, and you don’t know how you’re going to sort through it all and look for more. The solution: stop looking. Go through what you have. If you find what you need in what you already have, great! If not, you can always keep looking. You don’t need to find everything in the first pass. There is plenty of opportunity to do more if more is needed!

From Theory to Practice

Looking back, the “Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End” case is more complicated than it may have first appeared. Scholars found evidence of fraud in several other published studies.^[4] These mistakes happened despite a formal editing process that should have ensured better accuracy and a peer review process that should have evaluated the material more closely. With both of these processes having failed, it was up to other scholars to figure it all out.

Now that you know more about the theory behind evaluating sources, it’s time to apply the theory. The following section will help you put source evaluation into perspective using something you probably remember from elementary school – the five Ws and H. You’ll also have the opportunity to try out your new skills with several hands-on activities.

The Five Ws

You evaluate sources regularly in your day-to-day life. You evaluate which movie to see this weekend, which new restaurant to try out, and which section of a required college class to take next semester. Evaluating sources in an academic context requires many of the same skills that you already employ.

When you begin evaluating sources, what should you consider? There is no single test or checklist that you can use to determine if a source is trustworthy or appropriate. Evaluating sources requires critical examination of each source, as well as the context in which you intend to use that source.

When evaluating sources, one set of criteria that you should consider are the five Ws and H. These questions can help you begin to suss out not only the quality of your source, but whether it is appropriate for your intended use case.

Who

Understanding more about your information’s source helps you determine when, how, and where to use that information. Is your author an expert on the subject? Do they have some personal stake in the argument they are making? What is the author or information producer’s background? When determining who is behind your source, consider the following:

• Who is the author? What are their credentials?
• What is the author’s level of education, experience, and/or occupation?
• What qualifies the author to write about this topic?
• What affiliations does the author have? Could these affiliations affect their position?
• What organization or body published the information? Is it authoritative? Does it have an explicit position or bias? Is it scholarly?
• Was this information funded by a group or organization? If so, who funded it? Is there a conflict of interest or perception of bias?

What

Knowing the intended purpose of a piece of information is a key to evaluation. Understanding if the information has clear intentions, or if the information is fact, opinion or propaganda will help you decide how and why to use information. When determining what is the purpose of your source, consider the following:

• What information does the author present?
• Is the information clearly supported by evidence?
• Does the source have an obvious bias or prejudice?
• What information isn’t presented? Does the author omit important facts or data that might disprove their argument?
• Is the article presented from multiple points of view?
• Is the author’s language informal, joking, emotional, or impassioned?
• Is the source intended to inform, sell, persuade, or entertain?

Where

Determining where information comes from, if the evidence supports the information, and if the information has been reviewed or refereed can help you decide how and whether to use a source. When determining from where a source derives its information, consider the following:

• Where does the information in the article come from?
• Where can I find more information about the topic?
• Is the source well-documented? Does it include footnotes, citations or a bibliography?
• Is information in the source presented as fact, opinion or propaganda? Are biases clear?
• Can you verify information from referenced information in the source?

When

One of the most important and interesting steps to take as you begin researching a subject is selecting the resources that will help you build your thesis and support your assertions. Many topics require you to pay special attention to how current your resource is—because they are time sensitive, because they have evolved so much over the years, or because new research comes out on the topic so frequently. When evaluating the currency of an article, consider the following:

• When was the item written and how frequently does the publication it is in come out?
• Is there evidence of newly added or updated information in the item?
• If the information is dated, is it still suitable for your topic?
• How frequently does information change about your topic?
• Has the information been updated recently?

Why

As you consider each source, consider the sheer quantity of information available to you. A simple Google search may retrieve billions of search results in response to a keyword search on your topic. Why is this source the best one to use for your particular topic and project? Remember that you’re not looking for the first source you can find on a topic – you’re looking for the one that is going to provide the strongest evidence for your information needs. When considering why you should use a particular source, consider the following:

• Why is this information better than other available sources?
• Why is this information considered high quality, scholarly, or trustworthy?
• Why does this article support or undermine your research hypothesis?
• Is the information written clearly and free of typographical and grammatical mistakes? Does the source look to be edited before publication? A clean, well-presented paper does not always indicate accuracy, but usually at least means more eyes have been on the information.

How

Understanding what resources are most applicable to your subject and why they are applicable can help you focus and refine your thesis. Many topics are broad and searching for information on them produces a wide range of resources. Narrowing your topic and focusing on resources specific to your needs can help reduce the piles of information and help you focus on what is truly important to read and reference. When determining how a source may be relevant to your topic, consider the following:

• How does this relate to what you already know?
• How do you intend to use this information?
• Does the item contain information relevant to your argument or thesis?
• Does the information presented support or refute your ideas?
• If the information refutes your ideas, how will this change your argument?

Conclusion

When you feel overwhelmed by the information you are finding, thinking through the 5Ws and H can help you determine which information is the most useful to your research topic. Remember that no test or checklist is guaranteed to tell you if a source is trustworthy, but considering these factors can help you better understand whether a source is appropriate for your topic and research project.