Academics have learned quickly that investigations based on data from online research agencies can have problems. Here are those problems and alternatives, according to Rice University researchers. Photo via Getty Images

Academics are learning quickly that investigations based on data from online research agencies have their drawbacks. Thousands of such studies are released every year – and if the data is compromised, so too are the studies themselves.

So it’s natural for researchers, and the managers who rely on their findings, to be concerned about potential problems with the samples they’re studying. Among them: participants who aren’t in the lab and researchers who can’t see who is taking their survey, what they are doing while answering questions or even if they are who they claim to be online. In the wake of a 2018 media piece about Amazon’s Mechanical Turks Service, “Bots on Amazon’s MTurk Are Ruining Psychology Studies,” one psychology professor even mused, “I wonder if this is the end of MTurk research?” (It wasn’t).

To tackle this problem, Rice Business professor Mikki Hebl joined colleagues Carlos Moreno and Christy Nittrouer of Rice University along with several other colleagues to highlight the value of other research methods. Four alternatives – field experiments, archival data, observations and big data – represent smart alternatives to overreliance on online surveys. These methods also have the advantage of challenging academics to venture outside of their laboratories and examine real people and real data in the real world.

Field experiments have been around for decades. But their value is hard to overestimate. Unlike online studies, field experiments enhance the role of context, especially in settings that are largely uncontrolled. It’s hard to fake a field experiment in order to create positive results since each one costs a considerable time and money.

And field experiments can yield real-life results with remarkable implications for society at large. Consider one experiment among 56 middle schools in New Jersey, which found that spreading anti-conflict norms was hugely successful in reducing the need for disciplinary action. Such studies have an impact well beyond what could be achieved with a simple online survey.

The best way to get started with a good field experiment, Hebl and her colleagues wrote, is for researchers to think about natural field settings to which they have access, either personally or by leveraging their networks. Then, researchers should think about starting with the variables critical for any given setting and which they would most like to manipulate to observe the outcome. When choosing variables, it’s helpful to start by thinking about what variable might have conditions leading to the greatest degree of behavior change if introduced into the setting.

Archival data is another excellent way to work around the limitations of online surveys, the researchers argue. These data get around some of the critical drawbacks of field research, including problems around how findings apply in a more general way. Archival data, especially in the form of state or national level data sets, provide information and insight into a large, diverse set of samples that are more representative of the general population than online studies.

Archival data can also help answer questions that are either longitudinal or multilevel in nature, which can be particularly tricky or even impossible to capture with data collected by any single research team. As people spend increasing amounts of time on social media, the internet also serves as a source of newer forms of archival data that can lend unique insights into individuals’ thoughts, attitudes, and behaviors over time.

With every passing year, technology becomes increasingly robust and adept at collecting massive amounts of data on an endless variety of human behavior. For the scientists who research social and personality psychology, the term “big data” refers not only to very large sets of data but also to the tools and techniques that are used to analyze it. The three defining properties of Big Data in this context include the speed of data processing and collection, the vast amount of data being analyzed and the sheer variety of data available.

By using big data, social scientists can generate research based on various conditions, as well as collect data in natural settings. Big data also offers the opportunity to consolidate information from huge and highly diverse stores of data. This technology has many applications, including psychological assessments and improving security in airports and other transportation hubs. In future research, Hebl and her team noted, researchers will likely leverage big data and its applications to detect our unconscious emotions.

Big data, archival information and field studies can all be used in conjunction with each other to maximize the fidelity of research. But researchers shouldn’t forget even more old-fashioned techniques, including the oldest: keen observation. With observation, there are often very few, if any, manipulations and the goal is simply to systematically record the way people behave.

Researchers – and the managers who make decisions based on their findings – should consider the advantages of old-style, often underused methodologies, Hebl and her colleagues argue. Moving beyond the college laboratory and digital data survey-collection platforms and into the real world offers some unparalleled advantages to science. For the managers whose stock prices may hinge on this science, it’s worth knowing – and understanding – how your all-important data was gathered.

------

This article originally ran on Rice Business Wisdom and is based on research from Mikki Hebl, the Martha and Henry Malcolm Lovett Professor of psychology at Rice University, and Carlos Moreno and Christy Nittrouer, who are graduate students at Rice University. Additional researchers include Ho Kwan Cheung, Eden B. King, and Hannah Markellis of George Mason University.

Ad Placement 300x100
Ad Placement 300x600

CultureMap Emails are Awesome

Houston-based health tech startup is revolutionizing patient selection for clinical trials

working smarter

On many occasions in her early career, Dr. Arti Bhosale, co-founder and CEO of Sieve Health, found herself frustrated with having to manually sift through thousands of digital files.

The documents, each containing the medical records of a patient seeking advanced treatment through a clinical trial, were always there to review — and there were always more to read.

Despite the tediousness of prescreening, which could take years, the idea of missing a patient and not giving them the opportunity to go through a potentially life-altering trial is what kept her going. The one she didn’t read could have slipped through the cracks and potentially not given someone care they needed.

“Those stories have stayed with me,” she says. “That’s why we developed Sieve.”

When standard health care is not an option, advances in medical treatment could be offered through clinical trials. But matching patients to those trials is one of the longest standing problems in the health care industry. Now with the use of new technology as of 2018, the solution to the bottleneck may be a new automated approach.

“Across the globe, more than 30 percent of clinical trials shut down as a result of not enrolling enough patients,” says Bhosale. “The remaining 80 percent never end up reaching their target enrollment and are shut down by the FDA.”

In 2020, Bhosale and her team developed Sieve Health, an AI cloud-based SaaS platform designed to automate and accelerate matching patients with clinical trials and increase access to clinical trials.

Sieve’s main goal is to reduce the administrative burden involved in matching enrollments, which in turn will accelerate the trial execution. They provide the matching for physicians, study sponsors and research sites to enhance operations for faster enrollment of the trials.

The technology mimics but automates the traditional enrollment process — reading medical notes and reviewing in the same way a human would.

“I would have loved to use something like this when I was on the front lines,” Bhosale says, who worked in clinical research for over 12 years. “Can you imagine going through 10,000 records manually? Some of the bigger hospitals have upwards of 100,000 records and you still have to manually review those charts to make sure that the patient is eligible for the trial. That process is called prescreening. It is painful.”

Because physicians wear many hats and have many clinical efforts on their plates, research tends to fall to the bottom of the to-do list. Finding 10-20 patients can take the research team on average 15-20 months to find those people — five of which end up unenrolling, she says.

“We have designed the platform so that the magic can happen in the background, and it allows the physician and research team to get a jumpstart,” she says.” They don’t have to worry about reviewing 10,000 records — they know what their efforts are going to be and will ensure that the entire database has been scanned.”

With Sieve, the team was able to help some commercial pilot programs have a curated data pool for their trials – cutting the administrative burden and time spent searching to less than a week.

Sieve is in early-stage start up mode and the commercial platform has been rolled out. Currently, the team is conducting commercial projects with different research sites and hospitals.

“Our focus now is seeing how many providers we can connect into this,” she says. “There’s a bigger pool out there who want to participate in research but don’t know where to start. That’s where Sieve is stepping in and enabling them to do this — partnering with those and other groups in the ecosystem to bring trials to wherever the physicians and the patients are.”

Arti Bhosale is the co-founder and CEO of Sieve Health. Photo courtesy of Sieve

Houston nonprofit unveils new and improved bayou cleaning vessel

litter free

For over 20 years, a nonprofit organization has hired people to clean 14 miles of bayou in Houston. And with a newly updated innovative boat, keeping Buffalo Bayou clean just got a lot more efficient.

Buffalo Bayou Partnership unveils its newest version of the Bayou-Vac this week, and it's expected to be fully operational this month. BBP Board Member Mike Garver designed both the initial model of the custom-designed and fabricated boat as well as the 2022 version. BBP's Clean & Green team — using Garver's boat — has removed around 2,000 cubic yards of trash annually, which is the equivalent of about 167 commercial dump trucks. The new and improved version is expected to make an even bigger impact.

“The Bayou-Vac is a game changer for our program,” says BBP field operations manager, Robby Robinson, in a news release. “Once up and running, we foresee being able to gain an entire workday worth of time for every offload, making us twice as efficient at clearing trash from the bayou.”

Keeping the bayou clean is important, since the water — and whatever trash its carrying — runs off into Galveston Bay, and ultimately, the Gulf of Mexico. The improvements made to the Bayou-Vac include removable dumpsters that can be easily swapped out, slid off, and attached to a dump truck. The older model included workers having to manually handle trash and debris and a secondary, land-based vacuum used to suck out the trash from onboard.

Additionally, the Bayou-Vac now has a moveable, hydraulic arm attached to the bow of the vessel that can support the weight of the 16-foot vacuum hose. Again, this task was something done manually on the previous model of the Bayou-Vac.

“BBP deeply appreciates the ingenuity of our board member Mike Garver and the generosity of Sis and Hasty Johnson and the Kinder Foundation, the funders of the new Bayou-Vac,” BBP President Anne Olson says in the release. “We also thank the Harris County Flood Control District and Port Houston for their longtime support of BBP’s Clean & Green Program.”