Chapter 6.

Data-Gathering and Research

By the end of this chapter, students will be able (1) to understand the distinction between quantitative and qualitative data-gathering methodologies, (2) to understand the data-gathering principles and techniques presented in the chapter, and (3) to formulate questions for use in an interview or a questionnaire in order to gather information for developing hypothesis to explain problematic behavior.

Teaching Notes

Recommended Instructional Outline:

Lesson 6 consists of an interactive lecture session lasting about 2 hours. There is a homework assignment at the end of the lesson.

The lecture is as follows:

1. Review of Lesson 5 (15 minutes). Review Lesson 5 objectives. Review Lesson 5 homework. Resolve any outstanding questions the students may have from Lesson 5.

2. Preview of Lesson 6 (15 minutes). Preview Lesson 6, using the chapter outline above.

3. Lecture (1 hour and 30 minutes). The main lecture portion will teach students (1) to understand the distinction between quantitative and qualitative data-gathering methodologies, (2) to understand the data-gathering principles and techniques presented in the chapter, and (3) to formulate questions for use in an interview or a questionnaire in order to gather information for developing hypotheses to explain problematic behavior.

3. Introduction

In the last several chapters, we have introduced a problem-solving methodology for solving social problems. Along with presenting the methodology, we have also provided a sample social problem to analyze and propose solutions to. To facilitate understanding, we have provided all the materials you needed to complete each step of the process. In real life, of course, you will not be given all of the materials, conveniently organized for you. You will be required to conduct your research using newspapers, scientific studies, scholarly works, and other such sources of information. When you do not have access to such information, you will may be forced to conduct research yourself. This chapter will introduce you to the fundamentals of gathering data to explain problematic behavior and to test your hypotheses regarding this behavior.

4. Research and gathering data generally

A draft policy or law is not based on abstract logical conclusions, but on logical conclusions grounded in facts and experience. Whether the drafter relies on the experience of others (in the form of formal research or anecdotal reports), or on the drafter’s own experience (based on formal research or observation) an understanding of research methodology and data gathering is crucial to obtain accurate data for use in formulating effective policy.

With respect to the four steps of the problem-solving methodology, a drafter needs to organize facts and data in order (1) to describe the nature and scope of the social problem or difficulty, (2) to analyze and explain the causes of the problematic behavior of the role occupant and the implementing agency, (3) to test hypotheses about the problematic behavior, to develop alternative solutions, and to choose the best solution, and (4) to conduct a cost-benefit analysis that evaluates the economic and social benefits, as well as the negative impacts, of a proposed policy or law.

The search for relevant data can be focused in two areas: (1) data obtained from existing sources, and (2) data obtained through stakeholder participation.

(a) Gathering data from existing sources

The drafter of a policy or law is seldom directly involved in gathering original or new data. Generally, through interviews with knowledgeable individuals or from existing documents and information, a drafter performs “research” by collecting and organizing facts that have already been gathered by others.

(b) Gathering data through stakeholder participation

Data gathering often tends to focus on “expert” opinion regarding a phenomenon, whether it be a scholarly article or an interview with a government official with specialized knowledge of the problem. It is important for the drafter to involve stakeholders — not only as the object, but also as the subject, of research — in order to promote a more democratic process of policymaking and lawmaking.

There are a number of reasons to include stakeholders in research. First, stakeholders have interests in the proposed policy or law, so they have motivation to cooperate with the researchers. Second, stakeholders have their own knowledge about the related problem and the environment in which the problem occurs, so they can participate in gathering facts in the four steps of the problem-solving methodology. Finally, involving stakeholders in research enhances their capacity to cooperate in finding a solution to social problems.

5. Data-gathering methodologies

This section discusses the two methods for gathering data: (1) quantitative methodology, and (2) qualitative methodology.

(a) Quantitative methodology

Quantitative methodology, as the name implies, focuses on how a phenomenon expresses itself in objective quantity. (For example, age can be expressed in years, height can be expressed in meters, price can be expressed in rupiah, and so forth.) In social sciences, quantitative methods generally involve calculations related to people and their thoughts or opinions. The advantage of using a quantitative methodology is that the results of the inquiry can be easily compared. (For example, you might determine that the average age of students in Makassar is higher than the average age of students in Denpasar.) The disadvantage is that by breaking down a phenomenon into units for easy comparison, the data may have limited use. (For example, knowing the average ages of students in Makassar and Denpasar tells us nothing about their intelligence, motivation, or likelihood of future success.)

(b) Qualitative methodology

Qualitative methodology focuses on how a phenomenon is perceived in subjective quality. (For example, the beauty of a sculpture, whether the price in rupiah is too expensive, and so forth.) This methodology encourages the participation of stakeholders, since stakeholders are most likely to have information regarding the social problem.

6. Application of data-gathering methodologies to the problem-solving methodology

This section shows how both quantitative and qualitative methodologies can be used in each of the four steps of the problem-solving methodology, as described below.

(a) Step 1 — Describing the problem or difficulty

Describing the social problem is the first step in advocating change. As an effective advocate you need to express both the quantity of the problem and its quality. In other words, “How much is it happening?” and, “How much does it hurt people?”

(b) Step 2 — Analyzing the problematic behavior

Each ROCCIPI factor can be thoroughly examined only by considering both quantitative and qualitative data. Note that some ROCCIPI factors or categories (such as interest and ideology) may rely on one type of data more than the other type. (In the case of interest factors, quantitative data is usually more relevant. In the case of ideology factors, qualitative data is usually more informative.)

(c) Step 3 — Testing hypotheses and proposing solutions

The type of data used to explain the causes of the problematic behavior will mirror the type used in creating hypotheses. For example, if a qualitative analysis indicates that a “buyer beware” attitude is an underlying cause of shop owners selling defective products, a qualitative search for solutions (through interviews, etc.) may result in a possible solution.

(d) Step 4 — Conducting a cost-benefit analysis

A complete cost-benefit analysis includes both quantitative and qualitative data. Costs and benefits can be measured in terms of both quantity (for instance, an amount of money) and quality (for instance, safety, enjoyment, or loyalty).

7. Techniques for gathering data

This section discusses three techniques often used by researchers for gathering data: (1) one-on-one interviews, (2) first-hand or direct observation, and (3) questionnaires or surveys.

(a) Interviews

Researchers very often obtain data or information from people who have knowledge of some aspect of the problem. They do this through one-on-one interviews. The person being interviewed is called the “respondent”. Interviews may be conducted face-to-face (in person), by telephone, or even over the Internet. However they are conducted, the basic concepts for an effective interview are the same. The two basic types of interview types are (1) structured interviews, and (2) unstructured interviews.

(1) Structured interviews

A structured interview uses a list of questions to be asked by the interviewer. The interviewer communicates the questions to the respondent and may be able to ensure that the respondent understands the questions. In many ways this technique resembles the questionnaire (discussed below), with the added advantage of having an interviewer available to explain any uncertainties or ambiguities the respondent may find in the questions.

(2) Unstructured interviews

An unstructured interview is one in which the interviewer obtains information about a specific subject based on guidelines rather than a list of specific questions. These guidelines consist of the main points or ideas about the matters that are to be asked in the interview. In an unstructured interview, the interviewer has freedom in how to formulate questions to obtain the desired information about the main points or ideas described by the interviewer’s guidelines. Using the guidelines, the interviewer is free to ask questions and to probe more deeply and follow-up important matters in order to enrich the information (data) obtained.

The unstructured interview can be divided into two types: (1) focused interviews, and (2) free interviews.

(A) Focused interviews

A focused interview is one that aims to clarify a research problem with a hypothesis that has been formulated beforehand, even though more specific questions about the concepts in the hypothesis have not been formulated, or detailed in writing, in advance. This technique is basically like a “half-structured” interview.

(B) Free interviews

A free interview has no structure at all. It mostly depends on the development of the question-response process between the interviewer and the respondent throughout the interview. The interview is conducted with few, if any, basic guidelines concerning its content or how questions should be formulated.

(b) Direct or first-hand observation

Researchers can also gather data through direct or first-hand observation of people or situations. This observation enables researchers to explain behavior or test hypotheses.

When observing people’s behavior, the subject of the observation, must be unaware that his or her behavior is being observed, lest the respondent artificially alter the behavior being observed. (For example, an ankot driver that knows that you are studying ways of reducing traffic jams may avoid stopping at unsanctioned stops in order to give you the impression that such behavior is not an issue.)

(c) Questionnaires or surveys

Researchers also gather data using questionnaires or surveys in which respondents answer a series of questions (usually in writing). When using questionnaires or surveys, the respondent plays a very important role in helping the researcher obtain the needed data. Using a questionnaire is similar to conducting a structured interview (discussed above), but the questions are posed in writing to the respondent.

When conducting a survey, the questionnaire may be answered by many different methods, including by using (1) pen or pencil and paper, (2) a computer program, or (3) the Internet.

Questionnaires may be given to respondents and returned to the researcher either directly or indirectly. Very often, questionnaires are sent and returned indirectly by mail. These are called “mailed questionnaires”.

There are several factors that may affect whether the respondent returns the questionnaire to the researcher. These include (1) who the researcher or research sponsor is, (2) the length of questionnaire, (3) the means by which the questionnaire is delivered to the respondent and to be returned to the researcher, and (4) any compensation provided to the respondent for participating in the research by completing the questionnaire.

8. Rules for creating questions to be included in an interview or a questionnaire

In order for the researcher to obtain primary data needed from the respondents, a data-gathering tool is required. A list of questions to be used in an interview or in a questionnaire is an important standard tool for gathering data. This section describes the rules for formulating effective questions and effective lists of questions.

(a) General rules for the overall list of questions

Before compiling a list of questions, the researcher needs to consider (1) clarity of concepts and related indicators, (2) standardization, (3) objectivity, and (4) the relevant unit or method of measurement.

(1) Clarity of concepts and related indicators

The researcher must determine clear boundaries of the concepts related to the behavior to be explained or the hypothesis to be tested. The researcher must then determine what informational indicators are related to the concepts identified. Data concerning the indicators is what the researcher needs to gather in the research. Therefore clear boundaries of the concepts will facilitate the researcher to compile a list of questions. Once a list of indicators has been determined for each concept, the researcher can design questions to obtain information about those indicators.

(2) Standardization

In order to compare information obtained from respondents with different backgrounds and experiences, questions must be standardized, to eliminate or account for such differences.

(3) Objectivity

Objectivity is important in obtaining valid results. When formulating your questions, they should be as neutral as possible, to avoid suggesting a certain response and to prevent introducing bias into the results. You should formulate your questions in manner that avoids different respondents having different understandings or interpretations of the same question.

(4) Relevant unit or method of measurement

The researcher must choose a unit or method of measurement that is appropriate to both the type of data to be obtained and the respondent being asked the question.

(b) Rules for formulating individual questions

When formulating questions to be included in an interview or a questionnaire, the researcher should consider (1) the type of question to be asked, (2) the form of the question, and (3) the content of the question.

(1) Types of questions

There are four types of questions that are generally used. They are (1) background questions about the respondent, (2) questions to elicit the respondent’s opinions or comments, (3) questions about the respondent’s level of knowledge or information, and (4) questions about the respondent’s perceptions.

(A) Background questions

Background questions about the respondent are those concerning facts about the respondent’s personal characteristics. These questions often include (1) residence or other geographical information, (2) gender, (3) age, (4) marital status, (5) occupation, (6) education level, (7) religion, (8) ethnicity, and so forth.

For example:

Question: What is your age?

Question: What is your current occupation?

Question: Have you ever participated in training concerning your profession?

(B) Questions eliciting opinions or comments

Questions may be designed to elicit the respondent’s opinions or comments about certain matters.

For example:

Question: What is your opinion about protected forest areas?

Question: Do you believe that shifting cultivation activity is destroying the forest?

(C) Questions on knowledge or information

Questions may be designed to measure how much knowledge or information the respondent has concerning certain matters. These questions reflect the respondent’s level of knowledge concerning the subject matter.

For example:

Question: How long have you known that there is a protected forest in the area?

Question: Do you know the advantages of a protected forest to your life?

(D) Questions on perceptions

Questions about the respondent’s perceptions measure how the respondent evaluates a certain matter in relation to another matter or person. There are no right or wrong answers to these questions. Through the answers about the respondent’s perceptions, the researcher will obtain an image of the subject matter from the respondent’s point of view.

For example:

Question: If you had financial capital, you would .................. (Respondent is to complete the unfinished sentence.)

Question: Do you prefer to borrow financial capital from (1) a bank, or (2) an individual?

(2) Form of questions

There are three forms of questions that are generally used. They are (1) closed questions, (2) open questions, and (3) half-open questions.

(A) Closed questions

A closed question provides a respondent with a specific and finite list of possible alternative responses or categories of answers. Respondents simply choose one of the alternatives or categories of answers provided, according to their choice. A good closed question is one that provides alternatives or categories of answers that (1) are complete, and (2) do not overlap with one another. Note that sometimes the respondent may choose more than one answer (often these questions contain special instructions like, “Choose one or more of the following:”).

For example:

Question: How much money do you earn in a month?

Answer:

(1) < Rp. 100.000.

(2) Rp. 100.025 – Rp. 200.000.

(3) Rp. 200.025 – Rp. 300.000.

(4) Rp. 300.025 – Rp. 400.000.

(5) Rp. 400.025 – Rp. 500.000.

(6) > Rp. 500.000.

Question: What factor most motivates you to cut down trees?

Answer:

(1) The question does not apply to me because I do not cut down trees.

(2) I need the wood for my own cooking or firewood.

(3) I need money obtained from sellling the wood to others.

(B) Open questions

An open question does not provide a specific list of possible responses or categories of answers. Respondents are free to give their own answer, according to their own will.

For example:

Question: How much money do you earn in a month?

Answer: Rp. ....................

Question: What factor or factors motivate you to cut down trees?

Answer:

(1) ....................

(2) ....................

(3) ....................

(C) Half-open questions

Sometimes the researcher wants to suggest a list of possible responses but also allow a different answer from the respondent. This form of question provides one or more alternatives or categories of answers but also lets respondents answer according to their own will.

For example:

Question: Which type of trees do you think are most useful as reforestation plants?

Answer:

(1) Teak trees.

(2) Mango trees.

(3) Other (specify): ....................

Question: What factor or factors motivate you to cut down trees?

Answer:

(1) I need the wood for my own cooking or firewood.

(2) I need money obtained from sellling the wood to others.

(3) Other (specify): ....................

(3) Content of questions

The important concern in formulating the content of a question is that the question should be clear and simple. A question is clear if the respondent can understand the meaning of the question asked. A question is simple if it has only one meaning and does not lead to another interpretation. There are several considerations in formulating the content of a clear and simple question. These are discussed below.

(A) One matter per question

In formulating a question, try not to put two matters that you wish to ask into one question, as in the commonly known “double-barreled” question. Instead, ask two questions.

For example:

Do not ask this “double-barreled” question:

Question: When and where did you first hear about the reforestation program in your area?

Instead, ask these two questions:

Question 1: When did you first hear about the reforestation program in your area?

Question 2: Where did you hear first about the reforestation program in your area?

(B) Avoid ambiguous or vague terms

In formulating a question, avoid words that are unclear or that have a vague meaning.

For example:

Question: Do you often catch fish in the Komodo National Park area?

In this example, using the word “often” can lead to different interpretations by respondents. One respondent may consider catching fish three times a month to be “often”, while another respondent may not consider three times a month to be “often” enough.

(C) Avoid using jargon or technical terms

In formulating a question, avoid jargon or technical terms. In general, avoid using words or language that the target respondent may be unable to understand or that do not meet the respondent’s capacity. Such words or language may cause the respondent to be unable to understand the meaning of the question put forward.

For example:

Respondents in a remote village, who generally have low education levels, find difficulty in understanding questions such as the following:

Question: How do you participate in the agricultural innovation process?

In this example, the terms “participation” and “innovation process” are abstract or technical terms that may be seldom used in rural areas.

(c) Order of questions

The last important matter in compiling a list of questions is the order of the questions. The order of the questions can affect whether or not the researcher obtains the desired data.

(1) Ask closed questions first, ask sensitive questions later

It is better to begin with questions (1) regarding non-sensitive subjects, (2) that are posed in a non-threatening manner. These questions are typically closed questions. Beginning the questioning with questions of a sensitive nature may lead the respondent to refuse further questioning, leaving you with many unanswered questions. Although what constitutes subjects that are sensitive depends on individual cultural preferences, the form of certain questions, regardless of their substance, may lead to respondent discomfort.

Generally, open questions require a respondent to reveal a personal preference as opposed to choosing from a set list. As such, open types of questions should be saved for later in the interview or questionnaire. Conversely, you should begin with closed questions, when possible, before moving on to open questions or sensitive questions.

(2) Use “filter” and “follow-up” questions

Some questions are really composed of two parts: (1) a “filter” question, and (2) a “follow-up” question. The answer to the first part (the “filter”) determines what will be asked in the second part (the “follow-up”). For instance, in order to ask a question that involves only a few people in a certain category, you need to ask an initial “filter” question to determine whether the respondent is one of the people that fits into that category. If the person fits into the category, then you can ask the “follow-up” question to obtain the desired information.

For example:

Do not ask this “unfiltered” question:

Question: What kind of job do you do?

In this example, the “unfiltered” question presumes that the respondent has a job, which may not be the case. It is better first to ask whether the respondent has a job, then to ask what the job is. Thus, use “filter” and “follow-up” questions:

Filter question: Are you currently working?

Answer: ( ) Yes ( ) No

Follow-up question: If yes, what kind of job do you do? If no, why are you not working?

(3) Put questions in a logical order

Questions should be put in a logical order to avoid confusing the respondent. Questions jumping from one matter to another will break the respondent’s concentration in answering the questions. Conversely, after all the questions concerning a certain matter are asked, the questions may then move on to other matters.

9. Homework assignment

1. Write a questionnaire with ten questions, in which you use each of the three question forms outlined in this chapter. The questionnaire should be targeted at ankot drivers and should gather information relevant to at least four of the ROCCIPI factors.

2. Write a one-paragraph plan for using the observation technique to gather information about the problem of traffic jams. You may want to start by answering the following questions: Where would you go? Who would you observe? What aspects of the problem would you measure?

10. Further reading

The following materials provide further information about the issues discussed in this chapter and may be referred to for additional information.

[English version:] Ann Seidman, Robert B. Seidman, and Nalin Abeysekere, Legislative Drafting for Democratic Social Change (English version), Kluwer Law International, Boston, Massachusetts, 2001. Pages 167–186.

[Indonesian version:] Ann Seidman, Robert B. Seidman, and Nalin Abeysekere, Legislative Drafting for Democratic Social Change (Indonesian version, 2d ed.), ELIPS II National Library, Jakarta, 2002. Pages ___–___.

NOTES:

1. For the English version of this manual, please use only the reference to the English version of the Seidmans’ book (above).

2. For the Indonesian version of this manual, please only use the reference to the Indonesian version of the Seidmans’ book (above).

3. Please do this for all the chapters.

Ann Seidman, Robert B. Seidman, and Nalin Abeysekere, Assessing Legislation: A Manual for Legislators, online at http://www.bu.edu/law/lawdrafting/manual/, 2003. Chapter 8.

Earl R. Babbie, The Practice of Social Research (2d edition), Wordsworth Publishing Company, Inc., Belmont, California, 1979.

Masri Singarimbun and Sofian Effendi (editor), Survey Research Methodology, LP3ES, Jakarta, 1989.

Manasse Malo and Sri Tristoningtias, Community Research Methodology, PAU–IIS, Universitas Indonesia, Jakarta.

NOTES:

1. Please correct any of the non-Seidman references.

2. Please add references to any other appropriate (Indonesian) materials.

2. Please do this for all the chapters.