Competent Info Search in the WWW

Gerjets, P. & Hellenthal-Schorr, T.  (2008) Competent information search in the World Wide Web:  Development and evaluation of a web training for pupils.  Computers in Human Behavior 24(3) 693-715.

Also part of that special sectin in CIHB, this study focuses on 6th grade students in Germany.

Intro notes:  Students take the availability of Internet-based information resources for granted. As of 2004, 95% of 14-19 yr olds use the Internet.  As of 2001, 99% of US public schools had Internet Access.  Characteristics of the WW that present challenges:  open access for info retrieval and authorship, vast differences in user population, distributed authorship of docs, lack of central agency responsible for structure/quality of docs, lack of stability, vast amount of info, broad nature of conents, unordered network structure, and interaction of multimedia components.  Because of this, web users need skills such as orientation and search planning, info selection, and info evaluation to benefit from WWW resources.  Current approach of short-term Internet trainings, which focus on technical aspects and neglet cognitive aspects of internet searches.  Authors developed CIS-WEB (competent information search in the world wide web); aims at improving cognitive and metacognitive aspects of students' ability to search the www.  Following 2 theories guided development:

1.  Conceptual analysis of info search from media literacy/info retrieval research:

• media literacy rsch tends to list sub-competencies, independent of specific medium and users' goals.
• Info Literacy Perspective (Doyle):
• recognizes that accurate and complete information is the basis of intelligent decision making
• recognizes need for information
• formulates questions based on information needs
• identifies potential sources of information
• develops successful search strategies
• accesses sources of information (incl electronic)
• evaluates info
• organizes info for practical application
• integrates new info into an existing body of knowledge
• uses info in critical thinking and problem solving
• Media Literacy sub-competencies (Gapski):
• technical knowledge of how media work
• practical knowledge of how to use media
• self-reflective w/ regard to relation between media usage and env.
• creative skills
• social-reflective (how media influence social resp/political action
• coping w/ media effects.

• Most important content aspects for info search in WWW (schorr):
• background knowledge in terms of development and structure of Internet as information environment.
• skills for using computers (connect to internet, browser software, search engines)
• Ability to keep oriented in regard to info sources provided by www
• ability to evaluate info provided in www in terms of relevance in context of current info problem, quality, and credibility.  Ability to select info according to those criteria.

•  info retrieval tends to identify sub-processes. (more user-oriented)
• Models (blair, clark, eisenberg & berkowitz, kuhlthau, marchionini)  segmented into sub-processes and propose appropriate sequence.  Five sub-processes most imortant:
• specification of info requirements
• application of search strategies
• handling search systems
• selection and evaluation of information/sources
• monitoring processes and results of information searches

Pulling from both lines of research, there are five integrated content aspects the researchers used as the basis of developing CIS-WEB:

1. Representation of WWW as an information environment: media background knowledge and operation skills with regard to www.
2. specification of information requriements:  segmenting of info problems into sub-problems, sequencing of information sub-problems, formulations of hypotheses in terms of type and localization of information needed.
3. application of search strategies:  typing in urls, following links, using search engines, distinguishing appropriate info for different search strategies.
4. selection and evaluation of information and information sources:  evaluating relevance, quality, credibility, and actuality of information in context of current info problem as well as the selection of info according to those criteria
5. monitoring of info searches:  knowledge on monitoring of processes and results of info searches.

Also developed systematic taxonomy of different search strategies based on problem-space paradigm by Newell and Simon (1972).  Tasks distinguished according to their complexity and how to adapt search strategies.

Design of CIS-WEB:  five pivotal content aspects plus four sub-goal structures of info.  Six training modules developed.  Problem-based training.  All six modules are covered in a total of 12 45 min lessons.  Contents:

1. Representation of WWW as info environment:
1. definitions, services, origin, user  stats, access options of internet
2. different formats of info, contents, authors, lack of quality control, security advice, continuous change of info, technical terms, web-browsers.
3. def of search systems, functionality of different systems, selection criteria for different search systems.
2. Info problems, sub-goals, and complex sub-goal structures:
1. categorizing info problems according to sub-goal structure; info requirements for different categories of info problems
2. basic knowledge on how to adapt search strategies to sub-goal structures
3. intro to 4 pivotal sub-goals of info on www:  locating info on website, locating website, selecting info provider, id of subtasks.
3. Locating info on Website
1. typical structure of website
2. using search funcitons and sitemaps as web tools
3. strategies for quickly assessing contents of websites
4. evaluating relevance of info in context of current info problem
4. locating website
1. direct retrieval (urls, inferring urls)
2. following links
3. searcy systems:  portals and search engines, selection of keywords, definign queries, selecting links in query results.
5. Selecting an info provider
1. eval of credibility of info source
2. eval of actuality of an info source
6. ID sub-tasks of info problems (breaking down complex info problems into sub-tasks and selecting strategies to help with sub-tasks)
1. breaking down complex info problem into subtasks and specifying different info requirements.
2. iding operators/methods to achieve sub-tasks
3. monitoring processes and results of info searches

Instructional Methods used in CIS-WEB:

Instructional setting:

• regular classroom teaching:  concepts introduced and discussed using powerpoint
• working in student pairs based on a computer-based hypermedia environment.
• individual work: exercises and problem-solving assignments based on paper-pencil materials, corrected either in pairs or by whole class with help of teacher.

Instructional methods:

• Elaborated worked-out examples:  example info problems from different content areas with a step-by-step solution to convey details of finding solution to info problem.
• use of symbols to illustrate complex and abstract relationships (structures of Internet) and to demonstrate important procedures for an info search.
• Interactive mult-choice questions w/ feedback
• worksheets: paper/pencil based problem-solving assignments forcing students to apply new knowledge.
• recapitulations:  at beginning of each new module, contents of previous models were reviewed; at end of training, entire module was reviewed.

Study 1:  tests common assumption that attending short-term Internet training is sufficient. ("surfcheck-online")  2 hypotheses:

1. participating in a conventional Internet training (cmpared to an unguided explorationof the web) will increase pupils' search-relevant and search-irrelevant declarative knowledge.
2. participating in a conventional Internet training will improve pupils' search performance when trying to solve information problems with the help of the WWW, at least if these information problems are not too complex with regard to their sub-goal structure.  

Method:
participants:  28 6th grade pupils from public German school (21 female, 7 male), avg age 11.92 yrs.
materials and procedure:  3 subsequent days, 2 45-min lessons per day.  Day 1: pretesting.  15 min to fill in declarative knowledge test w/ 20 mult-choice items, 75 min to solve 1 of 2 sets of 16 info problems each.  Questionnaire on computer/Internet experience.  Day 2:  implement instructional intervention--"surfcheck online" or unguided exploration of www).  Day 3:  posttest.

Results: 

All pupils had used computers at school or at home for at least 1 year.  92% used Internet--73% for at least one year, 50% at least once a week.  92% reported they were experienced with info retrieval on www.

Prior declarative knowledge for search-irrelevant facts 73.1%, search-relevant facts 66.9%.  However, solving info problems 17.2% correct.

Post-knowledge for "surfcheck" students:  search-irrelevant gained 4.55%, search-relevant gain .80%
Post-knowledge for independent exploration students:  negative gains in both areas.

Hypothesis 1 not proved.

Search performance decreased (-4.46%)
Hypothesis 1 not proved.

Summary of study 1 results:  pupils face problems searching the web, which increase when search tasks become more complex.  Neither conventional Internet training nor unguided exploration improve search performance or declarative knowledge with regard to search-relevant and search-irrelevant facts.  A more comprehensive training is needed.

Study 2:  tests effectiveness of CIS-WEB

Hypothesis 1.  Search compentence on the www increases with age.  thus, 8th graders should outperform 7th graders.  These differences should disappear after CIS-WEB due to training effects.
Hypothesis 2.  CIS-WEB improves pupils search-relevant and search-irrelevant declarative knowledge with regard to the web.
Hypothesis 3:  Pupils' search performance improves due to studying CIS-WEB.  Search performance is expected to increase for all four sub-goal structures although information problems of more complex sub-goal structures are expected to result in an inferior overal search performance than information problems of simpler sub-goal structures.
Hypothesis 4:  The time course of pupils' search performance improvements depends on the complexity of the information problems.  This assumption is based on the modular structure of the web training where the knowledge necessary to solve information problems of more complex sub-goal structures is conveyed later than the knowledge necessary to solve less complex search tasks.  Consequently, search performance regarding information problems with more complex sub-goal structures is expected to only increase substantially in later phases of the web training.

Participants:  61 pupils (7th and 8th grade) from a public German high school (30 female, 31 male).  Average age 12.74.  avg age 7th grade 12.33, avg age 8th grade 13.3.

Materials/procedure:  3 subsequent days; 12 lessons of CIS-WEB plus six additional lessons needed for testing.  Six lessons on each of 3 days.
Day 1: declarative knowledge test and module 1 of CIS-WEB, plus 10 info problems (same test as study 1).  Day 2: declarative knowledge test and modules 2-4, plus 10 info problems.  Day 3:  modules 5-6, declarative posttest, 4th and 5th sets of info problems.


Results: 

All pupils used computers in school or at home (at least 1 year, 93%.  98% used Internet--at least 1 year, 71%, at least once a week: 58%.  88% report experienced w/ info retrieval on WWW. 

Prior declarative knowledge:  search irrelevant facts 70.5% correct, search relevant facts 60.5%.  Problems olving info problems:  18% correct.

First hypothesis:  no siginficant effects of pupil grade on declarative knowledge test or search performance at beginning or ending of training.  Not proved--age does nto necessarily lead to more competent information searches.

Second hypothesis:  declarative knowledge increased after 1st module of CIS-WEB, but between module 1 and end of course of study, pupils forgot some of the search-irrelevant facts so there was a decrease of declarative knowledge for them.  Search-relevant fact gains remained.  CIS-WEB had a positive impact on declarative knowledge acquisition.

Third hypothesis:  pupils' search performance on all 4 sub-goal structures improves while using CIS-WEB.  More complex sub-goal structures had lower performance than simpler sub-=goal structures (as expected). 

Summary:  CIS-WEB substantially improves declarative knowledge and increases performanceon solving information problems of different sub-goal structures.  Module 1 had the strongest training effects.  Modules 2-6 relied heavily on pupils engagement with worksheets/exercises, so motivation may have impacted results.    There are substantial correlations between pupils' learning intensity (performance on worksheets/exercises) and later learning outcomes--only pupils who were actively involved in working on module 2-6 benefitted substatially from those later modules.  However, suggestions for improvement of CIS-WEB include changing instructional methods in modules 2-6 to increase active participation, extend module 6 to improve learning outcomes of "identification of sub-tasks," which might lengthen web training.

Overall summery and discussion: 

Options for improving CIS-WEB:  focus on module 1, which improved performance substantially, and implement it in place of current web training. OR optimize instructinal methods in CIS-WEB to ensure more engagement in models 2-6, extending training materials and spacing presentation over a longer period of time (maybe 2 lessons per week for several weeks).  This might avoid pupils getting tired or overwhelmed and increase willingness to invest in worksheets and exercises, or use group work and small problem-based activities to enrich the course.  Also interweave CIS-WEB with other school subjects to students can apply skills to academic work. Also broaden target group to include senior citizens or other user populations.  Finally, improve technical tools and innovations to conform with how people conduct searches (ie, you could tackle this problem by changing the search engines rather than changing the user habits).  However, since current technology cannot undertake metacognitive processes, this system-oriented approach could only go so far and some user training would still be required.

My thoughts:  One concern I have with studies like this is that they seem to focus on one unit of study, as if you can just teach this once and everyone will "get" it.  It seems to me, based on my experience and the austrailia study I abstracted, that in order for students to internalized these strategies and use them regularly (rather than being able to tell you what the SHOULD do but not actually doing it), they need repeated exposure to these concepts over an extended period of time.  In other words, teach the specific modules, yes, and integrate them with coursework, yes, but also teach the classroom teachers to review the concepts regularly and expect students to demonstrate their use of them repeatedly in research assignments, over several grade levels/subject areas, using consistent steps and vocabulary.  Also, as technology changes, the course will have to be updated and teachers will need additional training.