Reflection on theme 6: Qualitative and case study research

I enjoyed reading the articles this week, and in particular my case study article. As case studies generally are hands on and easy to grasp it made it easy to understand the idea of the studies and the methods used during the conversations we had during the seminar, as well as when reading peer blog posts.  

It feels like we discussed the qualitative research methods a lot during the quantitative theme, so it was interesting to read one of those papers, and to see the difference in the output. In my case study paper which was partly based on interviews, I felt like I got to “meet” the test subjects and hear their opinions and struggles which made it very personal. This was partly due to the fact that the topic was for them to reflect on their own learning experiences and motivations.

I learned this week that the classification and coding process of the interview answers is an important and complex process. It seems as the researchers (at least in my article) go through a lot of work and several iterations to get these comments classified in a true and non-biased way, but the way this was  handled was difficult to follow in the paper.  

As my case study article was based on qualitative research it was a great opportunity for me to read two different qualitative research methods. They were very different, and the case study gave a lot more information, background and depth to their research. It was really interesting for me to see the quality difference and how much more it is possible to with a well-prepared and thought through method and analysis.   

We had an interesting discussion about the pros and cons of focus groups in our first seminar this week. I guess you could consider the seminar group a focus group as well, and I learned that this is a great way to get more creative and to reach further than you would individually. As it often is the case, there is no one great answer to what type of quality method to use when collecting your data; it all has to be adapted to the particular case you are working on. It takes a lot of effort and planning not to “disturb” the answers you get. Reading all the peer blog posts this week it was interesting to see what the researchers had done to get the test subjects to feel at ease, ranging from hiring an “expert interviewer”, to having a moderator and an observer during focus groups sessions, and even stating that they deliberately were not trying to act so authorial and trying to be at easy-going.  

To wrap up this course I have learned a lot about different research methods, and what to think about when conducting research. I have learned about the importance of how to ask your questions, and where to ask them (or not to) in order to try to get a wider audience for your survey base. I have also learned a lot about how to critically review a research paper, and how different it can be perceived by even among us students who are in many ways rather similar. 

I thought the blogs worked well in this class and I learned a lot from my peers reading their reflections, ideas and comments, as well as the comments on what I had written myself.  

Theme 6: Qualitative and case study research

  

The qualitative research paper I have chosen this week is written by: C Lister, J West, R Richards, B Crookston, P Cougar Hall, A Redelfs (2013), and it is called “Technology for health: A qualitative study on barriers to using the iPad for diet change”.

The study investigates if iPads are an effective tool when wanting to change somebody’s diet behaviors.  The authors used focus group (n=20) of college students who were all given an iPad for three weeks including some AppStore credit with the instructions to investigate apps (both free as well as pay-apps) in order to help them change their eating patterns. After the three week evaluation period, four focus group discussions were held.

The outcome of the study was that the iPad was conceived as too large to bring with you every day in order to report on your food intake. The focus group would have preferred a smartphone with a more convenient size, and which is something you anyway carry with you at all times. The apps were also perceived as either too intensive (cumbersome) or not intensive enough, which might indicate that a new design of these apps are needed.     

The qualitative method in this paper was four group discussions sessions, using two moderators. The main methodological issue with this study is that all test persons were college students, which likely means that they are comfortable with digital information and apps. This study was done on the first generation of iPads, and it seems as it was the first iPad experience for the test persons. However I think that there would have been a much larger threshold for older generation users to even find and start using these apps. Also it is not clear if the authors have made individual interviews, but it would have been constructive to do so. Some test persons might be affected of the views and ideas of the other people in the group. Test persons might also find it difficult to talk freely in a larger crowd.  

The case study I chose to read was: Lin, Liang, Yang, Tsai (2013), “Exploring middle-aged and older adults’ sources of Internet self-efficacy: A case study”.  

This study was about the perceived Internet self-efficacy (= självupplevd förmåga) of a group of older people trying to learn to use a computer and internet. It was designed as a set of instructional activities aimed at helping these novice computer users to get basic computer skills such as saving files, sending an email, and searching for information on the internet.  

Eisenhart (1989) defines in her article a nine step process on how to build theory from case study research, and below is my attempt to apply these steps my article.

Getting Started: The authors had a clear purpose and research question. They wanted to study what affects the Internet self-efficacy of a group of older people with low previous experience.  

Selecting Cases: The authors had a thought through approach on their instructional sessions, well adapted to their test group older people with limited computer experience. The test group was also well defined, middle aged people defined as 45-64 years old and older people defined as 65+ years old.

Crafting Instruments and Protocols: The data was collected by observation during the instructional sessions, as well as during individual interviews. The interview protocol was semi-structured and gave room for free answers.

  

Enfolding literature: The author’s uses many references in a clear and constructive way.

Analyzing data: The answers were analyzed and categorized in descriptive codes. Each code was assigned to one of the four different sources of Internet self-efficacy.  The method of applying these codes and categories is described in the paper, but was very hard for me to follow. They used qualitative features (observation during sessions) as well as the classified answers from the interviews for their analyses.     

Shaping hypotheses: The authors started off with the four sources of Internet self-efficacy proposed by Bandura (1986, 1997). During their study and interviews they were able to formulate a source which were not included in the theoretical construct and which seemed to be of great importance, namely proactive personality.

Reaching closure:  The authors conclude that the sources of self-efficacy seem to be more complicated than was previously assumed, and they were able to find a previously overlooked source of self-efficacy, the proactive personality. The findings also give way for a proposal in order to try to help people with low self-efficacy by trying to help them become more pro-active, which over time will strengthen their feelings and self-esteem.

Description of a case study: A case study is according to J. Geering (2004): “an intensive study of a single unit for the purpose of understanding a larger class of (similar) units”. In other words a case study is an investigation which aims to understand the dynamics within the settings of the study. Case studies are useful for forming general ideas and using a test set-up with all other things being equal, and the output is generally more ideal than concrete.

• Eisenhardt, K. M. (1989). “Building Theories from Case Study Research”. Academy of Management Review, 14(4), 532-550. (Impact Factor 7.895)
•  C Lister, J West, R Richards, B Crookston, P Cougar Hall, A Redelfs (2013). “Technology for health: A qualitative study on barriers to using the iPad for diet change”.  Health, 2013, Vol.05(04), p.761, (Impact Factor 1.137)
• Y-C Lin, J-C Liang, C-J Yang , C-C Tsai (2013). “Exploring middle-aged and older adults’ sources of Internet self-efficacy: A case study”.  Computers in Human Behavior (Impact Factor 2.067), Volume 29, Issue 6, November 2013.
•  John Gerring (2004). “What Is a Case Study and What Is It Good for?” American Political Science Review Vol. 98, No. 2 May 2004 (Impact Factor 3.933)

Reflection on Theme 5, Design Research

This week we have been focusing on design research. I enjoyed reading the articles this week as the topics were more hands-on with concrete and interesting design ideas.

The lecture with Ylva Fernaeus added depth to the article we read, as she explained their thinking and methods while doing this research study. I think their idea is really clever and could be developed further, especially when it comes to toys, which is a big market with a possible big money-revenue (especially at Christmas time, like when this is written). It made me think of a train set my son had when he was younger. They had developed little plastic (obviously) code bricks that made the train act on a message, realized by two raised bars of different length, which were detected by two press-buttons on the bottom of the train. 

Lego code brick that made the Duplo train stop

My son had lots of fun with these, making the train turn around, stopping and blowing its horn. Using small RFID-sensors built in to stackable code bricks could obviously allow for a bigger variety of interactions, and more detailed messages, maybe two or three bricks together could act as a combined by slightly altered action.

Ylva also had a very simple and concrete way of describing the several steps of research and knowledge. She phrased knowledge as something that you take notice of, which I think was a good description as well as very easy to understand. Your research is then to define this ‘something’ that you have taken notice of, and other people has perhaps not realized its importance or definition, and then to further explore this topic/idea. You then need to communicate and convince other people about the value and truth of this idea of which you have taken notice.      

The idea in Li et al’s article was also very innovative. I am not interested I sports, so I am not sure about this very application of their study, but I am sure that the sense modality can be used for a lot of message transferals in a near future. This will certainly be very important in our very noisy lives of today, not to disturb other people as well as receiving messages in the subway/busy street etc.  

Haibo Li held an entertaining as well as very interesting lecture. His lecture was much more engineering focused and we learned that we can all define a great idea (which he declared corresponds to a billion dollar industry) if we just focus on the real problem. It is also important to have an “elevator pitch” ready to communicate our idea to the right investor. Sounds really simple.

Li gave us a really illustrative example with a hungry bear, on why you need to focus on the “real” problem. The student realized that the answer to the hungry bear problem was that you do not need to outrun the bear, you only need to outrun the professor, as the bear will be happy with one lunch (in other words the professor, if the student was able to outrun him).

As my knowledge of design research before preparing for this week was very low I also learned a lot by reading the blog posts of my peers, as they had reflected on design research, prototypes, and the as Malin Westerlind defined the three factors affecting usability is effectiveness, efficiency and satisfaction.  

Overall, I think that the blog posts have been a very interesting way to increase my knowledge of theory and method during this course. It has been really interesting to read some very insightful comments about what all my peers have been thinking about on these various topics and the research articles they have chosen. 

Theme 5 - Design Research

A media technology research is often aimed at creating a new utility/software or a new feature of an already existing system. It needs to be evaluated by testing functions, and user perception. Questions that need to be answered are: is it feasible, useful, understandable, does it bring added value or “want-factor”? When doing testing and evaluation, it is also important to remember that the experience can be influenced by a resistance to new concepts amongst the users and that new technology often in the beginning is used as trying to resemble the familiar old technologies.

Design research is a very iterative process. It starts as a design concept, then a prototype is constructed, which is tested on functionality and characteristics, and then a better design concept emerges, the prototype is updated and new tests are done, and then a revised design concept is introduced etc. 

The prototype is very useful when it comes to evaluating the new design. Imagine creating new software without doing any coding at all, it would be hard for the users to give their input on those features. It is often not possible to create a fully functional device or prototype, especially in early studies, or it might not even be clear exactly how the new device should look like. The prototype is designed to be handled by the users in real life situations, and it helps them get a feel of the real product. 

An example from a school I heard about which had introduced a new laptop program and were about to offer all students a brand new individual laptop. The school decided to give all middle- and elementary school students a book of similar size and weight as the computer they were about to receive. The students were all expected to carry around this book (read prototype) at all times for a week before actually being handed their brand new and expensive laptops, paid for by the school. This way the students could learn how to handle the laptop in various situations (recess, lunchrooms etc).

The characteristics of the prototype should aim to help the users to understand the main features of the finished product. 

The example in the study of Réhman, S., Sun, J., Liu, L., & Li, H. (2008), regarding experiencing a football game through a mobile phone, the important characteristics were to show the size, shape and weight of the finished product and the main feature being the built-in vibrator. In this case it was not necessary to be able to make a phone call on the prototype mobile phone. 

In the study of Fernaeus, Y. & Jacobsson, M. (2009) the prototypes used was accessories and clothes with a built RFID tags (a tiny radio frequency ID-system) which altered the behavior of the dinosaur, vacuum-robot and the GlowBots.  By switching these accessories you could change the behavior of the robot (no programming involved – at least for the user..).  

The limitations of prototypes are that they are generally not fully functioning, they can be expensive to produce (small volumes), and they might turn out not to be sufficient for the study. It is likely that they need to be modified and improved over and over again as the design matures.

Design research can be presented in a flow chart with variables and their dependents. It can be communicated as a list of requirements, goals, design decisions and limitations. It can be presented through a display, trial and test-run of the prototype. It seems to me as a well-functioning prototype is an important tool for presentation and selling your idea/design, and therefore a great investment in the study.   

My question for the lecture is: I assume that design research is an iterative process – design solution – production of some kind of prototype  – testing of prototype - new design solution – new prototype – new testing – new design solution etc. When and on what grounds do you determine that your design is ready for a user/field study?

References:

Réhman, S., Sun, J., Liu, L., & Li, H. (2008). Turn Your Mobile Into the Ball: Rendering Live Football Game Using Vibration. IEEE Transactions on Multimedia, 10(6), 1022-1033.

Fernaeus, Y. & Jacobsson, M. (2009). Comics, Robots, Fashion and Programming: outlining the concept of actDresses. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. New York: ACM. 

Website: Complete Beginner’s Guide to Design Research, by UX Booth http://www.uxbooth.com/articles/complete-beginners-guide-to-design-research/, copyright 2008-2013, published on June 1^st, 2012, last updated (no date), retrieved on Dec 5, 2013

Theme 4, Quantitative Research: Reflection

This week’s topic was quantitative, as well as qualitative studies. After having been reflecting on the meaning of these theory-types, I now understand that I had wrongly categorized the article that I had chosen for the ‘Research and Theory’-week as a qualitative study (Wells, Valacich, Hess (2011):  “What Signals Are You Sending? How Website Quality Influences Perceptions of Product Quality and Purchase Intentions”, published in MIS Quarterly). I now realize that this was clearly a quantitative study, and that I guess is proof that I have learnt something about different types of theories during the course of these few weeks.

During the first seminar we discussed the articles that we had read, and we made a flow chart of the main variables in the theory. It was interesting to see how you can break down a study into concrete statements and dependents. We had a good discussion in our little study group, and it is again interesting to hear about the great variety of studies that have been chosen.

The most rewarding part of this week was the second seminar with Olle Bälter. He gave some good advice and pointers on how to conduct survey questions, for example: ‘never use NOT in your questions’, ‘make sure to avoid ambiguous questions’ and most of all ‘pilot your questions before sending them out!’.  His game-session was a great way to get the discussion going. It made us all think hard about quantitative vs. qualitative studies, as well as good and bad aspects of online vs paper surveys.

Olle Bälter mentioned that he had done a study (about 5 years ago) on the cost-effectiveness on paper vs. web questionnaire, and it surprised me to hear that up to 3000 surveys are more cost-effective to be distributed on paper (by post) rather than over the web.        

The medium on which you distribute your survey will affect how it is perceived, and it will affect the response-rate. It is important to note that it is not possible to say that one distribution type is preferable over the other, as it depends on the context and the topic of the survey. How you phrase your questions will affect the answers you get, and if you are conducting an interview – be sure to realize that the interviewer is affecting the outcome of the answers.

To sum this all up, we have learned this week that it is very difficult to perform a study that does not affect the outcome of the answers. It is important to think twice (or three times) about how to conduct your study in the least disruptive way. 

Theme 4: Quantitative Research

This week I have read the article “Facebook and texting made me do it: Media-induced task-switching while studying” by Larry D. Rosen, L. Mark Carrier and Nancy A. Cheever (2013).

In this study 263 Middle School, High School and University Students were observed during a study session of 15 minutes in their own home. They did this by educating 263 student observers, in how to take minute-by-minute notes on what the subject of study was doing (on-task studying, texting, browsing the internet, eating etc.). Each observer could chose up to three friends to observe while studying for a period of 15 minutes in his/her own home. The authors hoped to come up with strategies on how to cope with technical stress and how to make studying more efficient and on-task.  

All these minute-by-minute notes were collected, and by using regression analysis they could see several interesting patterns regarding study attitude and how likely a student would be off-task while studying during these 15 minutes. For instance: students, who had a preference to finish one task before moving on to the next task, were significantly more likely to be off-task while studying, and that students who checked their Facebook at least once during the 15 minute study period did in fact have lower grade-point average than the average study group person.

The limitation to this study is that they had to depend on the observers, who had no other experience than the received training session on how to take notes. The authors had no means to check whether notes were taken in a correct or uniform way.  It is also likely that the subjects who were observed tried harder than usual to stay on task during these 15 minutes of study as compared with if they would have been alone in their room. The benefit with being observed by a friend, and this was important for the study, was that the studies could take place in an environment as normal as possible for the students, and without feeling uneasy about being observed by a stranger.

In this paper, as well as the paper I read last week, I found it interesting how they excluded tests persons from the study and on what grounds they did so. I guess you have to exclude people, because not everybody will conform to the study or respond to questionnaires, it just had not occurred to me that you need to declare also those people in the research paper.  I also realized that I am not up to date with all the statistics calculations they perform.

As discussed above I think the main methodological problem in this study is that the test persons who are being observed while studying might not behave like they normally would, even though they tried their best to make them feel comfortable at home. If they had filmed them during a week with a video camera, it might be more realistic as the test people might forget about being observed. This however would make the study more expensive and it would be more difficult to collect all the data (like number of open pages on the computer during the study session).  

Quantitative methods:

In a quantitative research the goal is to determine the relationship between one thing (an independent variable) and another (a dependent or outcome variable). In my study above the independent variable was "distractions during studying" and the outcome variable the "grade point average".

The benefit of using a quantitative method is that it tries to control the study environment in order to isolate and recognize the variables in study. It aims to collect a large amount of data to study, and when it comes to study people, it tries to use a large and random group in order to have results which are representative over the entire population. The limitations lie in setting up the test case not tainted with outside-the-study variables, as well as finding a test group that is in representative of everybody.   

Qualitative methods:

Qualitative research method try to investigate the how and why of question, and generally investigates the human side to problems and peoples experiences. It generally generates rich, detailed data that leave the participants' perspectives intact. With this kind of study the difficulty is making sure your results are broad enough to make general recommendations. It is often more difficult to collect a large amount of data using this method, and you need to rely on a narrower field of view or results. The smaller data-references might make it more difficult detect false or fabricated and unrealistic answers.   

References:

• Fondell, E., Lagerros, Y. T., Sundberg, C. J., Lekander, M., Bälter, O., Rothman, K., & Bälter, K. (2010). Physical activity, stress, and self-reported upper respiratory tract infection. Med Sci Sports Exerc, 43(2), 272-279.
• “Facebook and texting made me do it: Media-induced task-switching while studying” by Larry D. Rosen, L. Mark Carrier and Nancy A. Cheever, published in Computers in Human Behavior (Impact Factor: 2.067) Volume 29, Issue 3, May 2013, Pages 948–958
• USC Libraries, Organizing your Social Sciences Research Paper, Quantitative Methods.   http://libguides.usc.edu/content.php?pid=83009&sid=615867 Last updated Nov 13, 2013, (retrieved on Nov 28, 2013)
• USC Libraries, Organizing your Social Sciences Research Paper, Qualitative Methods.   http://libguides.usc.edu/content.php?pid=83009&sid=615866, Last updated Nov 13, 2013, (retrieved on Nov 28, 2013)

Reflection on Research and Theory

The main focus during this week was “theory”, and on this topic we studied “The Nature of Theory in Information Systems” by Gregor (2006) as well as the paper with the unusual title “What Theory is Not” by Sutton & Staw (1995). During the two seminars this week we have reflected on the definition and use of “theory”, both in scientific ways and in daily use.

The concept of theory (in short being a proposition that is scientifically verified by empirical tests) might come across as not that complicated to understand, especially when compared to lots of other principles we need to digest as students of KTH. What makes it confusing, in my opinion, is the rather contradictory way we use the word theory in our daily life. When we say “I have a theory about this”, we imply that the idea we have is not backed up by any evidence about the underlying facts. Our daily use of “theory” more resembles the definition of a hypothesis, and it drains the “market value” of the scientific term theory.

In marketing I would compare this theory-confusion with brand names that have become everyday nouns, like “Thermos”, “Band-aid” or “Frisbee”. At a first glance it might seem like great PR for the product, but in fact it generally becomes a real trouble for the company who tries to advertise the original trademark as it is worn out by everyday use and confused with other products of lower quality, which all reflect back on the authentic and typically high quality product.    

Another thing I have realized this week is the importance of defining and limiting your research properly. In early research this might make us readers frustrated as it seems too simple, and they seem not to be able learn a lot by their experiments. One very interesting example we discussed during our seminar was an early study of vision-replacement by sound. A study was able to prove that blind test persons could match two socks with the same color, when they listened to a noise which varied with color. 

It was interesting to both read and talk about the large variety of research papers which have been studied this week. The variety of information and scientific research which actually is available online continues to amaze me. Maybe Adorno and Horkheimer were right, and we should worry about how much time people choose to spend looking at pictures of cute kittens, rather than “enlightenment”.  

References:

Gregor, S. (2006). ”The Nature of Theory in Information Systems.” MIS Quarterly, 30(3), 611-642.

Sutton, R. I. & Staw, B. M. (1995). “What Theory is Not.” Administrative Science Quarterly, 40(3), 371-384.

English Teacher Melaine, “Vocabulary – Xerox, Kleenex, and Roller Blades: When Brand Names Become Regular Nouns!”, Published on March 14, 2012, last updated (no date).  Available here: http://www.englishteachermelanie.com/vocabulary-xerox-kleenex-and-roller-blades-when-brand-names-become-regular-nouns/