Stage 1: Selection of Stimulus and Development of Questionnaire

As the study is to examine reaction to emotion-elicit provocative news, the first step was the selection of provocative news to use as stimulus. To do this, the research team first had to discuss and agree upon the definition of provocative news that was suitable for the study. As there is an absence of definitive and widely accepted definition of provocative news, the researchers draw on the limited literature that describes attributes of provocative advertisement[1]. Based on this as well as elaborate discussions between team members, the decided definition of provocative news is “news which attracts attention of the masses, touches on issues that are sensitive or deviates from cultural norms, elicit negative emotion, and could possibly instigate social disharmony and/or uncertainty”.

Based on the agreed definition, news that the researchers perceived as provocative was collected and a news pool was created. Using online news media (e.g. www.malaysiakini.com.my) as the source, news which had been shared within the social media, particularly in Facebook and WhatsApp were gathered within a period of one month (Feb/Mac, 2019). After omitting the news that were redundant and did not fit the agreed definition, a total of 47 news articles were identified and was subjected to further evaluation. Issues covered in the selected news include those that touches on race/religion and/or government affairs/performance. Stories covered by this news include specific events and announcements. 

Each of the news was rated based on perceived provocativeness by the four-member research team. They worked on the rating independently. Using a 10-point scale (from 1= low to 10= high), each news was assessed on the following criteria:  the news … (i) attracts attention of the masses, (ii) touches on sensitive issues, (iii) elicits negative emotions, and (iv) could possibly instigate social disharmony and/or uncertainty.

The team members then met to share and discuss results of their ratings. Based on the rating scores of each team member, the top-five news articles of each member were compared. Although the lists were not identical, there were similarities. The researchers had similar news in the top-five, but they were rank ordered differently.  Further discussions resulted in a decision to use two of the top-five as stimulus. The two news were validated in a pilot study. The two selected news were of different types. One is news about an event which touches on race/religious sensitivity (Stimulus 1), and the other is about government announcement which can be exploited by parties with political interest (Stimulus 2). To validate and ensure that the stimulus are effective, a single question asking participants to rate how provocative the news is (from 1= low to 10= high) was included in the survey questionnaire. Additionally, to validate the stimulus, an emotion elicit question was posed where participants had to indicate the extent to which they feel distress, upset and worried (from 1= not at all to 5= extremely) after reading the news.  

The next task within the first stage was the development of self-administered survey questionnaire. The specific items for scales and questions were extracted and adapted from existing relevant instruments available in the literature measuring the variables of interest in the study. The research team went through selected existing scales for possible inclusion.  Based on the reviewed and ensuing discussion, an initial pool of items of scales and questions were generated.  Where appropriate, adaptation was made to items to fit it to context. A total of 51 scale items and 18 questions including sociodemographic questions were developed for the initial questionnaire. The questionnaire was created in English and was then translated into Bahasa Melayu (Malaysia’s national language). The questionnaire was then subjected to arbiter analysis. Four social science researchers were invited to evaluate the appropriateness/suitability of the scales’ items and questions in the initial questionnaire.  For this purpose, an arbiter assessment form was created. It includes a description on background and purpose of the study, definition of constructs, instructions, and items of respective scales, and questions. Arbiters were asked to indicate yes or no on the appropriateness/suitability of each item and question. They were also asked to provide comments on items/questions they found problematic or unsuitable. Arbiters were also welcome to make any other comments or suggestions on the questionnaire. They worked independently on the analysis. To retrieve their comments, the researchers personally met two of them. The other two delivered their arbiter analysis entirely online. 

A number of improvements were made to the questionnaire based on the results of arbiter analysis. Overall, all arbiters agreed with the questionnaire items and questions. No item was identified as unsuitable. Nevertheless, there were comments that the wording on a few items were ambiguous or vague. For greater clarity and specificity, improvement was made on such items by changing words that were ambiguous or difficult to understand or by rephrasing the items/questions. The initial Likert scale items were on 5-point, from strongly disagree to strongly agree with undecided/neutral as the middle option. There was suggestion to change the middle response option to somewhat agree. The research team discussed on this, and eventually decided to use 6-point Likert item (strongly disagree, disagree, somewhat disagree, somewhat agree, agree, and strongly agree) instead of the 5-point response option There was one question in which two arbiters found the response option was inappropriate, and hence it improved accordingly. The final version of the initial questionnaire for the survey was in the national language (Bahasa Melayu). It was checked by a language expert. 


[1]Vézina, R., & Paul, O. (1997). Provocation in advertising: A conceptualization and an empirical assessment. International Journal of research in Marketing14(2), 177-192. DOI: 10.1016/S0167-8116(97)00002-5

Read more