Stage 2: Pilot Study

Purpose:

A pilot study was carried out from the 23rdto the 30th of April, 2019. Its purposes were to (1) determine whether the stimulus of the study was effective, that is the news was provocative and emotion eliciting enough as intended, and (2) assess the clarity of instructions, scales items and questions in the questionnaire, as well as (3) to determine internal consistency of the scales. The pilot study was also a way to get feedback from participants for questionnaire improvement purposes. 

Samples:

A total of 75 youths and adults (age range from 21 to 66 years, M=31.56, SD= 12.355) were recruited for the pilot survey. There were more females than males, 30% and 70%, respectively. In terms of ethnic group, 71% were Malay and 29% were non-Malay. Their participation was voluntary and they were given a small amount of money for their participation. The sample consisted of two groups of undergraduate students, two groups of graduate students, and two groups of UniversitiPutra Malaysia’s staff. They were from four different faculties. 

Data collection and instrument:  

Self-administered questionnaires were distributed in-person by the researchers. Participants were recruited from two undergraduate communication courses and one from a computer science course, two graduate communication courses, and the rest were university staff from three different faculties. Although administered in a group setting, each participant completed the questionnaire independently. On average, it took about 30 minutes to complete the questionnaire. In each data gathering session, half of the group was given questionnaire containing Stimulus 1, and the other half was given questionnaire containing Stimulus 2. The participants were first briefed on the general purpose of the study and how to complete the questionnaire. They were in no way primed that the study was about reaction to provocative news. This was to not sensitize them and influence their responses. Instead, they were informed that it was about social media news consumption-experience. They were encouraged to ask questions if they had problems understanding any of the items or questions. In the debriefing session, they were informed that the study sought to understand reaction to emotion-elicit provocative news they read in the social media. Participants were also asked for comments on the questionnaire. During the data collection sessions, the researchers checked the participants’ progress in completing the questionnaire. They were frequently given words of encouragement and were also reminded to read the items and questionnaire carefully. 

Thirty-four of the 75 participants completed the questionnaire which contained Stimulus 1 (provocative news of an event), and 41 participants completed the questionnaire which contained Stimulus 2 (provocative news of government announcement). There were three sections in the questionnaire. It started with a couple of question on frequency of getting and posting/reposting news on the social media, followed by a series of Likert scales items measuring the study’s variables (need for cognition; attitudes towards social media news sharing/discussion; subjective norm regarding reaction to problematic content in the social media, social media dependency, and news literacy which consisted of critical news consumption, media locus of control, and news consumption motivation). All the Likert-type items were a 6-point scale and they randomly ordered. This first section ends with a set of questions on personal motivational values. 

In the second section of the questionnaire, participants were asked to read a news (either Stimulus1 or Stimulus 2), and then asked to respond to a set of questions about the news and how they would react to it. The items measured their cognitive, affective, behavioral reactions as well as their perceived news salience. Participants were also asked a set of questions regarding methods they usually employ in verifying news that they receive on social media. The last section consisted of demographic questions which gauged their interest in politics and political affiliation.

Results: 

Overall, the questionnaire is acceptable. There were no major issues. Nevertheless, based on the feedback from participants retrieved during the debriefing/discussion session, a number of improvements or changes were necessary. Accordingly, the following modifications were made — (i) increased the word font size for easier reading, (ii) changed unclear or ambiguous words of several scale items for greater item clarity and specificity, and (iii) rephrased some items for greater clarity. 

All scales were subjected to an internal consistency test, and efficacy of the stimulus was assessed. With regard to the scales reliability, results of the analysis showed satisfactory reliability scores, the a-values range from .63 to .87. Nevertheless, the results suggest that there was room for improvement. In particular, as summarized in the Table 1, need for cognition scale, media locus of control, and critical prosumption each had one bad item that needed to be replaced. Deleting those items increased the a-values. In addition to replacing the bad items, one item was added to the each respective scale, predicting that this would increase theavalue of the scale in the actual data.  

Table 1: Reliability of scales 

With regard to the stimulus efficacy, results from a descriptive analysis on how provocative the stimulus was showed that Stimulus 1 and Stimulus 2 were effective. On a 10-point scale of provocative (1= low to 1= high), mean (M) for Stimulus 1 was 6.25 (SD= 2.489), and mean (M) for Stimulus 2 was 7.09 (SD= 1.616) (see Table 2). Therefore, both stimulus were retained and will be used in the actual survey. The decision to keep both stimuli is because they are of different categories. Stimulus 1 was about an event, and Stimulus 2 was about a government announcement.  It would be useful to compare the pattern of findings for the two stimuli. In the actual survey, participants will receive either a questionnaire containing Stimulus 1 or 2. 

Table 2: Descriptive summary statistics on degree of provocativeness 

In terms of the extent to which the stimulus was emotion-elicit, results from a descriptive analysis showed that indeed it was, thus adding further evidence on efficacy of the stimulus. On a 5-point scale (1= not at all, 2= slightly, 3= moderately, 4= very, and 5= extremely), the two stimulus were found to moderately invoke emotions of distress, upset, and worry. In other words, the selected news is to some extent an emotion-elicit provocative news.  Table 3 provides the relevant summary descriptive statistics.

Table 3: Descriptive summary statistics on degree of emotion invoked

Conclusion: 

The survey questionnaire was finalized based on the results of the pilot study. The final version contained 55 scale items and 27 questions including sociodemographic questions. Social desirability scale was also included in the final version of the questionnaire. The news selected as stimulus was retained. Participants will either receive a  questionnaire containing Stimulus 1 or 2 in the final survey.  

Read more