Digital Portfolio Performance Assessment at Summers-Knoll School

Abstract

Summers-Knoll School (SK) in Ann Arbor, Michigan, is an independent, kindergarten through fifth grade school with a focus on creative, multidisciplinary, project-based learning and authentic performance assessment via implementation of the University of Michigan-developed Work Sampling System (WSS), a comprehensive combination of narrative and portfolio measurement tools. This report documents research into the viability and efficacy of moving the current hardcopy portfolios onto a digital platform.

Participants in this study were teachers and staff at SK, a team of five graduate students at the University of Michigan’s School of Information (SI), and three graduate student volunteers at Eastern Michigan University. Teacher and staff interviews led to the development of a digital portfolio prototype by the SI team. Project funding constraints necessitated the search for already-existing digital portfolio software. The web-based Digitcation e-Portfolio application was located, tested on sample users, and analyzed. Results of the study indicate Digication’s software is relatively affordable, easy to use, and will meet the short to mid-term needs of teachers, students, and parents/guardians at SK.

I recommend SK license Digication e-Portfolio software for one year and implement a pilot program in the upper elementary/fourth and fifth grade class during the 2011-2012 school year. Following completion of the pilot, a benefits analysis should be conducted by the lead teacher or digital portfolio project manager to determine whether a whole-school roll-out should follow in fall 2012.

Introduction

Background: Summers-Knoll School

This is a graduate research project on written communication practices and contextual influences at Summers-Knoll School (SK) in Ann Arbor, Michigan. SK is an independent, kindergarten through fifth grade school serving bright, creative, and gifted children and their families. Founded in 1996 by Ruth Knoll and Jean Navarre, women who identified the need for a school dedicated to bright, creative, and gifted children (“History,” 2010), Summers-Knoll has evolved from an intimate educational environment located in Ms. Navarre’s basement into a fully accredited independent school. Summers-Knoll meets the needs of gifted and other exceptional learners through delivery of individualized, teacher-developed curricula that allow students to achieve their personal best in a minimally competitive environment. Basic skills are taught and reinforced by being imbedded in broader contexts, and unique talents of students, faculty, and staff are embraced and celebrated. Through heavy family involvement, Summers-Knoll takes a community approach to education that requires full-time commitment from teachers, students, and parents.

While many communication studies have been conducted on social and active approaches to K-12 student writing instruction and development, such studies have almost exclusively focused on student-based learning. Teacher-based communication research has been limited to the six language arts — listening, talking, reading, writing, viewing, and visually representing — and how these concepts are threaded by educators into the broader curriculum. Indeed, it has been my experience that the bulk of undergraduate teacher preparation is heavily skewed toward the idea of the K-5 teacher as a passive, interchangeable participant in the scientific delivery of pre-determined content using research-based instructional methodologies and measuring achievement using government-mandated, score-based assessment.

The reason for this heavy focus on student achievement and how to measure it may be due in large part to falling scores in reading, writing, and math, and the near-extinction of once-traditional elementary school staples such as history, geography, fine art, and penmanship (Meier, Kohn, Darling-Hammond, Sizer, and Wood, 2004). After a steady 20-year decline in student scores, particularly for minority and multi-racial students as well as those living in urban areas (Mills & Hamilton, 1994), the art and science of teaching has been seemingly handed over to the “experts.” The role of contemporary teacher has been reduced to disciplinarian, attendance taker, and worksheet distributer. Further, teachers themselves are faced with increasing administrative demands to obtain and maintain certification and constantly update credential files, and they must bear the brunt of public and government criticism for student failure. Within the K-12 education system, due to teacher tenure and labor unions, it has become increasingly difficult for districts to hire unique talent and dismiss teachers who are burnt out or underperforming. If we add to these woes the low prestige and pay of this profession (Mills & Hamilton, 1994), the current state of K-12 teaching is a bleak one indeed.

Though I do not claim to have a fix for all of education’s ills, as a college of education graduate and a ten-year teaching veteran, six years of which I spent as a homeroom teacher at SK, I do have insight into what, from my observation, contributes to the best and brightest in the field: autonomy. That autonomy is obtained by teachers becoming active creators instead of passive participants, and this is the cornerstone of Summers-Knoll School’s teaching and assessment philosophy, and SK’s assessment practices serve as the foundation for this graduate study.

I served as a homeroom teacher at Summers-Knoll School from 2002 to 2008. All homerooms are multi-age with a maximum size of twelve students. During my six-year tenure, I taught all core subjects — the language arts, social studies, mathematics, science — to students in grades one through five. I spent one year teaching history, geography, and English to children in sixth and seventh grades (the middle school has since disbanded), and I worked on special projects with kindergartners during my final year of employment. I enjoyed all of the grades, and, though I discovered a preference for teaching fourth and fifth grades, one of my most exciting and successful years was teaching a gaggle of precocious and quirky first and second graders. In addition to homeroom subjects, students receive instruction by content experts in a variety of classes known as specials. SK specials include Spanish, French, Latin, fine art, physical education, and music. Additionally, electives courses are offered to students in subjects such as Mandarin Chinese, theater arts, environmental science, horseback riding, chess, Greek, and gardening. These electives courses are taught by teachers and other content experts culled from the greater Ann Arbor area.

Teaching at Summers-Knoll School was an entirely gratifying employment experience. I had wanted to teach since being a first grader myself, but as an adult I was turned off by many schools’ standards and methodologies. After several years of substitute teaching and earning necessary credentials, I decided to pursue other work. I quickly changed my mind, though, after I happened upon Summers-Knoll. SK believes in teaching to the individual rather than to the median. SK celebrates divergent thinking with a focus on outcomes rather than processes. Most important to me, though, is the school’s empowerment of students and teachers. Students are expected to respond critically during learning, and teachers are called on to teach in the classical sense: to research, develop curricula, and deliver instruction creatively.

Measurement of student performance at SK is done through implementation of the University of Michigan-developed Work Sampling System (WSS). WSS is a two-pronged assessment tool consisting of narrative reports and portfolios. Though a variety of technologies are integrated into instruction at SK and relied upon by teachers and administrators for completion of WSS, the final student assessment products — narrative reports and portfolios — currently exist solely in hardcopy. Further, although hardcopy portfolios have served SK community members well, much of the project-based content created by students at SK is increasingly multimedia. Current short-term solutions for inclusion of such works in portfolios include web-based presentations and multimedia DVD slideshows of certain projects. A more comprehensive, multimedia-capable portfolio solution is needed by SK, however, and such a solution is the primary focus of this study.

Though SK continues using WSS’s narrative and portfolio assessment package to measure student achievement and growth over time, the school is now considering moving both components of the UM-designed tool onto digital platforms. WSS offers a web-based alternative to the paper and pencil model of its narrative product. Management of student data can be streamlined, archived, and reviewed allowing greater ease-of-use for cross-referencing and report generating. And it is my belief that moving narrative reports online will provide more meaningful student performance data that will satisfy the minority of SK parents/guardians seeking greater quantification of their children’s achievements.

WSS does not offer a web-based portfolio tool, though a variety of affordable software applications exist, the use of which would allow SK to craft a sort of a la carte digital assessment package based on the original structure of WSS’s narrative and portfolio model. Digital, or e-portfolios, have lately dominated much of the discussion of secondary and post-secondary student assessment. Their use in K-5 contexts has received minimal coverage, however, and the efficacy of their implementation at Summers-Knoll School is the foundation of this research study. Following is a brief survey of contemporary scholarship on digital or e-portfolios in K-12 settings. See Appendix A for a more thorough examination of WSS and written communication practices at Summers-Knoll School.

Literature Review

Though my project focuses specifically on digital, or electronic, portfolios, I expanded my reading to include the use of traditional portfolios in K-12 and post-secondary education as well. Foremost in my review was searching for a clear explanation of what a portfolio is. Ocak and Ulu offer a concise definition of the primary-level portfolio calling it “a carefully selected collection of student work that provides clear evidence to the student, parent, and…educators of the student’s knowledge, skills, strategies, grasp of concepts, attitudes, and achievement in a given area over a specific time period” (2009, p. 28). Peacock, Gordon, Murray, Morss, and Dunlop provide a digital-specific definition in their 2010 study: “An ePortfolio is an electronic system that facilitates the development, collection, and management of digital resources which may be drawn from a range of learner experiences over a period of time and could include those from formal and non-formal learning opportunities” (p. 828). This interpretation may be especially applicable to experiential and cross-curricular learning by suggesting that digital, or ePortfolios, are capable of capturing development and achievement that occurs outside of conventional scenarios, including, as Wall, Higgins, Miller, and Packard point out, art, physical education, design, and technology (2006). Perhaps the most succinct definition comes from Meyer, Abrami, Wade, Aslan, and Deault who call the electronic portfolio “a digital container capable of storing visual and auditory content including text, images, video, and sound” (2010, p. 84). By including the digital portfolio’s multimedia capabilities, the authors highlight another key benefit of electronic performance showcasing.

In searching for clear-cut descriptions of digital portfolios that would help me develop my own informed understanding, I was struck by the lack of consistency in terminology across the literature. Alternately referred to by Chang as web-based learning portfolios (WBLP) (2001), Meyer et al. as electronic portfolios (EP) (2010), and Vermilion as Efolios (2008), the naming or development of a lexicon appears secondary to charting and tracking the primary functions and purposes of digital portfolios.

The documented benefits of digital portfolio assessment are numerous. Among the most interesting to me and relevant to my graduate work with SK is Chang’s assertion that portfolio assessment identifies student skills rather than shortcomings, identifies the unique needs of each student, increases student motivation, and encourages students to develop decision-making skills (2008). This is in stark contrast to more traditional summative assessment tools prevalent in lecture/test models that focus on memorization and context-specific performance. Chang’s research also indicates that digital portfolio assessment may be beneficial to students with low motivation and/or self-esteem, underperforming students due to at-risk-classification, and special needs students including twice exceptional and gifted learners (2008). Wall et al. further suggest that digital portfolios are an appropriate method for recording achievement of pupils who find writing difficult (2006). In other words, portfolio assessment may meet the unique needs of learners whose talents sometimes fall outside the confines of the bell curve.

Lam and Lee’s research study focused specifically on how digital portfolio assessments can facilitate quantitative achievement in English as a second language (ESL) and the language arts. After implementation of portfolio assessment in the authors’ ESL classroom, the data indicate that students’ writings were more grammatically and structurally accurate and that students generated more and better ideas in their assignments (2010). Vermillion reports anecdotal evidence that digital portfolios improve skills in technology, public speaking, and leadership (2008). And Hewitt’s seminal 1995 research study on portfolio assessment, as reported by Chang, uncovered the following seven benefits:

  1. Portfolios demonstrate students’ growth
  2. Portfolios encourage students to set up learning goals
  3. Portfolios provide evidence of students’ efforts
  4. Portfolios demonstrate students’ performance
  5. Portfolios help teachers review students’ performance
  6. Portfolios stimulate students’ introspective thinking and enhance self-assessment
  7. Portfolios encourage students’ learning interests, improve their communication skills, and build self confidence (2001, p.436)

While all of the above provide obvious advantages to learners and teachers alike, the stimulation of introspective thinking is especially interesting to me. One of the precepts of contemporary education studies is that reflective learning results in increased retention and achievement; therefore one can draw a direct correlation between Hewitt’s list of benefits and authentic, measurable learning. This is further reinforced by Meyer et al. who posit that metacognitive activities, such as reflecting while building and maintaining digital portfolios, teach students to learn how to learn and overcome deficiencies in core competencies (2010). And such egalitarian learning, write Ocak and Ulu (2009) and echoed by Seitz and Bartholomew (2008), results in increased student empowerment in classrooms where teachers are facilitators and students are constructors of knowledge.

Research Methods

Graduate Project Inception

In fall 2010 I was invited by the head of school at SK to participate in a joint study and evaluation of the school’s performance assessment tools. Specifically, the school was interested in determining the benefits of moving the two-tiered assessment package of narrative reports and portfolios onto digital platforms. I was asked to work with a team of graduate students from the University of Michigan’s (UM) School of Information (SI) who were designing a digital portfolio prototype. My research would inform the prototype design and provide feedback and recommendations for future iterations of the digital portfolio product.

Participants

Partners and participants in this research endeavor were head of school at Summers-Knoll School; three homeroom teachers at SK; JSTOR Interface Developer and SK web developer; and a team of five graduate students at the University of Michigan’s School of Information.

Research Design and Procedures

The research began with pre-planning meetings with SK head of school and faculty, SK web developer, and the University of Michigan graduate student team. Two brainstorm sessions with all partners and participants took place in October and December 2010. The UM SI team were familiarized with SK’s use of portfolio assessment, informed of the unique needs of the SK community, and brought up to speed with current research and developments in digital portfolio development and implementation. At these meetings, several SK teachers expressed concerns that any benefits of moving portfolios onto digital platforms would be overshadowed by the amount of work involved in training teachers, students, and parents/guardians to use the new software. A sense of “if it ain’t broke don’t fix it” permeated much of the early teacher-driven discussion, and rightly so as SK’s assessment preparation and development is already time consuming and sometimes stressful for teachers.

Follow-up to these initial meetings included presentation to SK faculty and staff of the UM SI student-designed digital portfolio prototype, including whole-group usability testing and a question and answer period.

In January and February 2011, I completed hour-long interviews with SK’s head of school and three homeroom teachers. Each participant was questioned about her philosophy of education, work-related written communication practices, thoughts on current and potential future assessment practices at the school, and how digital portfolio assessments might affect her instruction, evaluation, and communication practices.

Finally, I conducted independent reviews of UM SI’s digital portfolio prototype as well as two already-existing, web-based software products: WSS Online and Digication e-Portfolios. The latter, a relatively inexpensive digital portfolio application, I loaded with sample content and conducted usability tests with graduate student volunteers in March through May of 2011. In the following section, I will discuss the findings and my research.

Results and Discussion

Pre-Planning Meetings

Initial meetings began in September 2010 between the UM SI team and SK’s head of school. These meetings determined the need for developing a digital portfolio assessment tool for use by the SK community, and the UM SI team agreed to develop the prototype that they would in turn use as a component for their own graduate portfolios.

I was invited to join the team effort in late September 2010, and I attended my first meeting in October. Having already worked closely with two homeroom teachers and the head of school at SK provided me with a level of comfort and familiarity I might otherwise not have had. At our first meeting, I shared with attendees my findings thus far on digital portfolio assessments, including their minimally documented application in kindergarten through fifth grade (K-5) settings. After the teachers expressed their concerns that developing and implementing digital portfolios might be superfluous and ultimately unsustainable, the assembly moved forward with brainstorming key components of a successful, meaningful, and user-friendly digital portfolio product. From this initial meeting, it was determined that the final product must have the following capabilities:

  • Ability to display multimedia content
  • Ability to graphically track student performance over time
  • Ease-of-use for students, parents/guardians, and teachers
  • A hierarchy of user categories/capabilities with teachers maintaining ultimate control of content
  • Multiple content sorting capabilities, i.e. by project, discipline, date, etc.
  • Inclusion of a student reflection component for each portfolio entry
  • Ability to easily link across entries within portfolios
  • Ability to print hardcopies of portfolios
  • Multi-year archiving capabilities
  • Secondary communication capabilities such as generating email lists, allowing subscriptions, and tagging

In addition, teachers suggested that the product beta testing and whole-school roll-out be accompanied by appropriate staff and student training and on-going support.

The next meeting of all partners and participants took place in December 2010. At this meeting, UM SI students presented a digital portfolio prototype based on recommendations generated from the October meeting as well as earlier discussions with the head of school. Following are three screen shots of the prototype.

Figure 1: Digital portfolio prototype login page

Figure 1 above is the digital portfolio prototype login page. Here is where the three primary portfolio users — teachers, students, and parents — can login to the web-based system, each with varying degrees of administrative rights. For example, students can login and add multimedia and text content to their portfolios. Parents can login and view and comment on only their child’s portfolio or portfolios of other children from whom they’ve received invitations. Teachers can login to a master view of all of their students’ portfolios and also approve or veto any student or parent submission prior to publication.

Figure 2: Digital portfolio prototype media entry and reflection page

Figure 2 above is a sample portfolio edit page accessible from all three user group portals. From here, teachers and students can upload media, teachers can add a narrative assessment, students can write a reflection, and parents/guardians can add comments. From this page users can also view a description of the entry (added by student or teacher), see a student image or avatar, and add and view tags associated with the entry. All content on this page must be approved by the teacher before final publication.

Figure 3: Digital portfolio prototype settings page

Figure 3 captures the digital portfolio settings page. From here, students and teachers can choose preloaded themes for portfolios or upload new themes. Portfolio fonts can also be indicated here. From this page, portfolio modifiers (i.e. students and teachers) can choose to close the portfolio for editing, making it inaccessible to other users/viewers while under construction. One of the most important digital portfolio features, specified during the initial brainstorming sessions with SK teachers and UM SI design team, is featured on this page, too. The Create PDF button allows users to download the entire student portfolio into a PDF file for emailing, printing, and local, digital filing. How such a feature will capture multimedia content had not been determined at the time of this prototype release.

The prototype was generally well-received by all partners and participants, and a whole-group usability analysis was conducted at this December meeting. Unfortunately, some of the prototype features, such as its suggested media drag-and-drop capabilities, were beyond the skill set of SK’s web developer. It was also determined, based on interviews with SK’s web designer and an additional Ann Arbor-based web development business, that contracting production of a digital portfolio based on the UM SI team’s prototype would be a more costly endeavor than was originally anticipated. To remedy this, I began searching for appropriate grant endowments that might offset the cost of software development. In early winter 2011, I applied for education/technology grants through AT&T, W.K. Kellogg Foundation, and W.L.S. Spencer Foundation.

Staff and Faculty Interviews

Between January and February, I conducted hour-long interviews with SK’s head of school and three homeroom teachers. All classes at SK are mixed age/grade. Teacher A, with more than 20 years teaching experience, taught a kindergarten and first grade (K-1) class. With more than 10 years of elementary education experience, Teacher B taught a second and third grade (2-3) class. And Teacher C, in her second year of teaching, served as homeroom teacher in the fourth and fifth (4-5) grade classroom. I asked each teacher to describe her teaching experience in detail, to express her philosophies of teaching and assessment, to explain communication practices at SK, and to provide a detailed analysis of SK’s two-pronged WSS assessment tool: narrative reports and portfolios. These meetings were relaxed, casual, and friendly. Further, they provided the teacher participants with a platform for honest expression of their ideas, thoughts, and concerns about the potential shift in assessment practices at SK. Though each expressed concern that the shift might be superfluous, all agreed to participate enthusiastically in the digital portfolio implementation if it was decided that such a move made sense for the SK community. Further, the interviews seemed to allow the busy teachers opportunities for reflecting on their own pedagogical and communication practices, which nicely mirrored the sorts of student evaluations that are at the heart of SK’s performance assessments. The head of school and teacher interview designs, including sample interview questions, were preapproved by EMU’s College of Arts and Sciences Human Subjects Review Committee.

Most notably revealed during the teacher interviews was the lack of infrastructure necessary for supporting the proposed digital portfolio design and implementation. Though SK has wireless networking capabilities, and all teachers are equipped with necessary hardware, such as MacBooks, digital cameras, external hard drives, and photo scanners, the classrooms themselves remain relatively low-tech. The K-1 class has no desktop computers, the 2-3 class a mere three, and the 4-5 class relies on a mobile bank of six to eight MacBooks for its computing needs. While this might not at first glance seem to pose much of a problem — computer instruction at the K-1 level is not a top priority for teacher A or the parents/guardians of her students — this lack of student-accessible hardware is potentially highly problematic if students are expected to actively participate in construction and management of their own digital portfolios. It was during these initial interviews with homeroom teachers that I concurred the best plan for roll-out of the digital portfolios at SK might be to begin with a year-long pilot project in the 4-5 class followed by project evaluation and a whole-school implementation the following year.

Shifting Gears: The Plan Changes

The original project plan was to work jointly with SK’s web developer to begin design, construction, and testing of a unique digital portfolio software application based on the UM SI team’s prototype and informed by the interviews conducted with SK teachers and head of school. In early February 2011, SK’s web developer withdrew from the project for a variety of reasons including time and funds. During this time of uncertainty, I completed three grant applications and interviewed two Ann Arbor-based web developers. Additionally, I began searching for already-existing software that might provide a short to mid-term solution to SK’s digital portfolio needs.

My search for an affordable, existing, digital portfolio application led me to Digication e-Portfolios. Digication’s web-based e-portfolio application was built by two graduate students at Rhode Island School of Design who created the product to showcase their own work. So successful was their software that it’s now used by a variety of secondary and post-secondary educational institutions, including Brown University, Yale School of Music, and Boston University. Digication has also partnered with Google to deliver a free version of their software to K-12 and post-secondary schools subscribing to Google Apps, a suite of education-specific applications. In late February, I began an email correspondence with Kelly Driscoll, one of Digication’s founders, which led to full access to a demonstration version of the e-Portfolio product.

The full version of Digication’s e-Portfolio software, which comes with live support services, is a relatively intuitive application. After login to the web-based application, the primary user (teacher) can see all of his or her e-portfolios at a glance. From this screen, the teacher can keep track of and manage an entire classroom of portfolios.

Figure 4: Digication teacher login page

To create a new e-portfolio, the teacher simply clicks on the Create button in the My e-Portfolios tab. After the new e-portfolio has been created, the teacher is able to customize the portfolio template with a corporate or institutional header, such as SK’s logo. The next task is for the teacher to create portfolio sections and then pages within the sections. For example, I created an SK sample e-portfolio with the following sections and sub-section pages:

  1. Bio
  • About Me
  1. Academics
  • Reading & Writing
  • Math
  • Social Studies
  • Science
  • PE
  • French
  • Music
  • Art
  • Extension Blocks
  • Field Trips
  1. Extracurricular
  • Cello
  • Chess Club
  • Soccer
  • Community Service
  1. Community
  1. Sports
  1. Contact
  • Contact Form

Setting up this structure took about two hours, and I referred to Digication’s downloadable user guide PDF for assistance as needed. (My own, SK-specific getting started guide is included as a downloadable PDF in Appendix C of this report.) Portfolio sections and subsection pages can be modified, expanded, or individualized at any time by the portfolio designer and whomever is granted modification rights, such as students. Read-only rights can be granted to other users, such as parents of students, and limited modification rights can be granted to other users with publication of changes strictly under the jurisdiction of teachers. (This ultimate teacher control is an important feature requested by each of the three homeroom teachers interviewed for this project.)

Inside each e-portfolio’s subsection page is where teachers, students, and authorized users add content. Each separate content upload is referred to as a Module. For example, using the framework I designed for this project, I uploaded a JPG photograph and brief biography of a sample student and placed this module on the Bio page. Below is a screen shot of this page.

Figure 5: Digication e-Portfolio sample bio page

In the Academics section, on the Music page, I uploaded a student/parent square dancing video filed at YouTube. On the Reading & Writing page I uploaded JPG images and a downloadable PDF file of sample student writing.

Figure 6: Digication e-Portfolio sample academic page

Each Digication e-Portfolio entry has a comments section where parents/guardians, friends, and other invitees may leave feedback. Publication of comments is, again, the final decision of teachers. Users may also add tags to individual entries, which is crucial for SK teachers who have requested the ability to sort portfolio content based on a variety of non-discipline keywords. For example, a math entry in a portfolio may be tagged as math and also as part of a larger project, say architecture, a recent 2011 school-wide curricular theme at SK. By being able to sort e-portfolio entries across disciplines, themes, and topics, teachers will be able to more easily graphically represent multidisciplinary instruction and learning. This is a potentially valuable tool when sharing digital portfolios with parents/guardians at conferences and also when reflecting on student performance while writing narrative assessment reports.

Software Usability Testing

In February 2011, I loaded sample content into the demonstration version of Digication e-Portfolio software I had been granted access to. While many of the functions of the software were easy to figure out based on prior experience, such as uploading multimedia content and click and drag reordering of module entries, others were less intuitive. For example, adding the Summers-Knoll logo as the anchored header for my e-portfolio required quite a bit of trial and error, and I eventually had to refer to online software support manuals. After completing the sample e-portfolio, I shared it informally with the head of school in a face-to-face meeting and sent email invitations to browse the demo to the three homeroom teachers I had previously interviewed.

In March though May 2011, I met with three EMU graduate student volunteers for usability testing of the sample e-portfolio. I based my usability tests on a survey of usability literature I conducted as part of my fall 2010 coursework in Advanced Technical Writing under the guidance of Steve Benninghoff, Ph.D., and drawn from previous usability testing I conducted in fall 2010 on Summers-Knoll’s web site. For the Digication e-Portfolio usability test, I created a list of 20 functions ranging from easy, such as accessing the e-portfolio in a web browser, to more challenging, like uploading a PDF file to a specific module. I tested each sample user separately and asked each to complete all 20 tasks. I provided each task in writing and also read it aloud. Before beginning each task, I asked each user participant if s/he fully understood the procedure s/he was being asked to perform. A list of all 20 tasks is included in Appendix B of this report.

During each task performance, I rated the users on a scale of 1 to 4, with 1 meaning the user easily completed the task and 4 meaning s/he experienced difficulty or frustration. If a user participant completed a task easily without any assistance, s/he scored a 1. If s/he successfully completed the task with trial and error, s/he was generally awarded a 2. A score of 3 was assigned if s/he required assistance or further clarification from me. A score of 4 indicated teacher frustration and/or inability to understand or complete the task without my involvement. Each of the three usability tests lasted between 30 and 60 minutes.

Upon completion of the usability tests, I tabulated the mean scores of the three participants in two ways. First, I found the average score of each individual participant. User 1 scored 2.1, user 2 scored 1.6, and user 3 scored 1.1. These scores indicate that overall, each user’s performance fell within the easy to somewhat challenging range. Next, I found the mean score of all three participants for each task. For example, task 10 was Drag to reorder the contents of the Reading & Writing page, replace the media, and modify the text. On this task, user 1 scored 3, user 2 scored 2, and user 3 scored 1, making their average score 2. While user 1 experienced some challenges with this particular task, the mean score of all three users indicates that in a collaborative environment, all three users should be able to successfully complete this task and other software tasks requiring similar algorithms and levels of difficulty. The most challenging task I asked the users to complete, number 20, was Change the title and theme of the demo e-portfolio. Users 1 and 2 both required heavy assistance from me and scored 4. User 3 was successful through trial and error and scored 2. The overall mean score for that task was 3.3, which indicates software training may be called for to familiarize users with more complex tasks. Overall, the test data indicate that potential users can successfully master the Digication e-Portfolio software with training and ongoing support/assistance. See Appendix B for the complete test data spreadsheet.

Conclusions and Recommendations

When I began this project in fall 2010, the central question I sought to answer through my research was whether or not moving Summers-Knoll School’s hardcopy student portfolio assessments onto digital platforms would increase the efficacy of student performance measurement. My findings were to inform the development of an original digital portfolio software application developed in conjunction with a team of graduate students from the University of Michigan’s School of Information and SK’s contracted web developer. Over time, as detailed in the preceding study, my focus shifted to locating and testing already-existing software that would meet the needs and requirements of SK teachers. This change was due to SK budget constraints and time constraints of SK’s web developer. Based on the needs of SK faculty and staff as determined in pre-study meetings and one-on-one interviews, I located an affordable, already-existing, e-portfolio application developed by Providence-based Digication. Though Digication’s e-Portfolio software is not as functionality-rich as the prototype developed by the UM SI team, I believe, based on my use of the product and subsequent teacher testing, it will serve as an appropriate and cost-effective introductory and interim solution for SK.

Recommendations for Summers-Knoll School

Following are specific recommendations for implementation of the Digication e-Portfolio package.

  1. I recommend Summers-Knoll School obtain a one-year subscription for the minimum number of Digication licenses. This is 200 licenses for $1,400. Included in this fee is system customization, administration tools, teacher training webinars, and phone and web support services. (An SK quick start guide is included as Appendix C of this report.)
  2. I recommend Summers-Knoll School appoint an existing full or part-time employee or hire a new full or part-time employee to manage the digital portfolio implementation project. Having a single point-person for project management will ensure consistency among student e-portfolios and teacher training as well as streamline and maximize communication between SK and Digication.
  3. Based on one-on-one interviews with Summers-Knoll School’s homeroom teachers as well as SK’s technology budget constraints and infrastructure needs, I recommend the school pilot the Digication e-Portfolio project in the fourth and fifth grade class during the 2011 – 2012 school year. (If a new middle school class is launched, as has been discussed, then the project should be piloted in this senior-level class.) Such a pilot project would best serve the school in several ways including:
  • Allowing for intense focus among a small number of relatively sophisticated users
  • Providing opportunities for development of best practices prior to whole-school roll-out
  • Mastery of software by older students who in turn could participate in teaching the application to younger students in the future
  • Providing a more manageable implementation project with streamlined reflection and assessment
  1. Throughout the suggested pilot, I recommend the program manager record ongoing feedback from the primary students/users and parent/guardians including comparison of e-portfolios to traditional hardcopy versions.
  2. Upon completion of the first-year pilot program I recommend a thorough review by the program manager including consultation with primary project stakeholders. The program manager should make recommendations for continuation of the project and whether or not a whole or partial-school roll-out of the Digication e-Portfolio software should begin in fall 2012.
  3. I recommend Summers-Knoll School continue seeking monies for its technology fund through grant applications, academic study partnerships, and other development activities. A successful partial or whole-school roll-out of web-based digital portfolios for students requires upgrades to existing hardware and software.

Summers-Knoll School has long been on the vanguard of integrated, multidisciplinary, project-based learning, and the school’s authentic narrative and portfolio performance assessments, developed by the University of Michigan, have the ability to accurately capture and quantify student achievement and growth. It is my belief that implementation of digital portfolios at SK will not only streamline performance assessment and multimedia communication but maintain the school’s position as a leader in progressive education and team-based communication.

Project Conclusion

My initial project plan, as documented in the preceding pages, was to serve as a communication liaison between the various stakeholders in this endeavor: SK staff and faculty, SK web designer, UM SI graduate students, and faculty advisors at EMU. My project proposal was predicated on the plan that meetings among SK faculty and staff would inform the UM SI team’s design of the digital portfolio prototype. Subsequent interviews of SK faculty and staff in regards to WSS and the digital portfolio prototype would provide a foundational structure for an original software application to be designed by SK’s contracted web developer. Finally, early iterations of this product would be tested on teachers at SK and findings of these tests would provide feedback to the software developer.

Like many real-world projects, this one evolved and changed over time. First, the UM SI-designed prototype, though competent and on-target, represented an application that was beyond the fiscal bounds of SK’s budget. Second, while I was searching for funds to bankroll the project, SK’s web designer pulled out. Rather than drop the project altogether, I searched for existing software that would be relatively affordable for the school to license and also meet the basic needs of the primary teacher, student, and parent/guardian users. What this graduate research endeavor ultimately became for me was an exercise in project management and interdepartmental communication. The hurdles, challenges, and successes experienced were authentic representations of those that occur every day in dynamic, people-driven organizations where decisions are dictated by a variety of factors, including changing rosters of participants, end user needs, and monetary constraints. In fact, the many detours from the original project proposal ultimately benefitted rather than hindered the primary objectives of my written communication graduate project by moving the project beyond the confines of academic research into an authentic professional situation. The proposal template modifications provided a foundational, professional writing context by placing the study of portfolio assessment in a real-world, professional situation, one with constraints, challenges, and competing stakeholder agendas.

_______________________________________________________________________________________

References

Akpinar, Y., Simsek, H. (2007). Pre-Service Teachers’ Learning Object Development: A Case Study in K-12 Setting. Interdisciplinary Journal of Knowledge and Learning Objects, 3, 197-217. Retrieved from http://ijklo.org/all.html

Alshamari, M., Mayhew, P. (2009). Technical Review: Current Issues of Usability Testing. IETE Technical Review, 26.6, 402-406. doi:10.4103/0256-4602.57825

Bailey, B. (2003). Evaluating the ‘Evaluator Effect.’ Retrieved from http://webusability.com/article_evaluating_the_evaluator_9_2003.htm

Baron, C. L. (2010). Designing a Digital Portfolio (Second Edition). Berkeley, CA: New Riders

Berrill, D.P., & Whalen, C. (2007). “Where Are the Children?” Personal Integrity and Reflective Teaching Portfolios. Teaching and Teacher Education, 23.6, 868-884. doi:10.1016/j.tate.2007.02.002

Blakeslee, A., & Fleischer, C. (2007). Becoming a Writing Researcher. Mahway, NJ: Lawrence Erlbaum Associates, Inc., Publishers

Bui, Y. N. (2009). How to Write a Master’s Thesis. Thousand Oaks, CA: Sage Publications, Inc.

Chang, C. (2001). A Study on the Evaluation and Effectiveness Analysis of Web-Based Learning Portfolio (WBLP). British Journal of Educational Technology, 32.4, 435-458. doi:10.1111/1467-8535.00212

Chang, C. (2008). Enhancing Self-Perceived Effects Using Web-Based Portfolio Assessment. Computers in Human Behavior, 24, 1753-1771. doi:10.1016/j.chb.2007.07.005

Chang, C., Tseng, K.H. (2009). Use and Performances of Web-Based Portfolio Assessment. British Journal of Educational Technology, 40.2, 358-370. doi:10.1111/j.1467-8535.2008.00885.x

Cobus, L., Dent, V.F., Ondrusek, A. (2005). How Twenty-Eight Users Helped Redesign an Academic Library Web Site. Reference & User Services Quarterly, 44.3, 232-246. Retrieved from http://www.emich.edu.ezproxy.emich.edu/library

Dicks, R.S. (2002). Mis-Usability: On the Uses and Misuses of Usability Testing. Proceedings of the 20th Annual International Conference on Computer Documentation, 26-30. doi:10.1145/584955.584960

Evans, D., & Gruba, P. (2005). How to Write a Better Thesis (Second Edition). Carlton, Victoria, Australia: Melbourne University Press

Hartson, H.R., Andre, T.S., Williges, R.C. (2003). Criteria for Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction, 15.1, 145-181. doi:10.1207/S15327590IJHC1501_13

Hertzum, M., Jacobsen, N.E. (2003). The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human-Computer Interaction, 15.1, 183-204. doi:10.1207/S15327590IJHC1501_14

“History.” Summers-Knoll School. (2010, October). Web. Retrieved from http://www.summers-knoll.org

Krug, S. (2010). Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. Berkeley, CA: New Riders

Lam, R., & Lee, I. (2010). Balancing the Dual Functions of Portfolio Assessment. ELT Journal, 64.1, 54-64. doi:10.1093/elt/ccp024

Lankshear, C., & McLaren, P. L. (1993). “Critical Literacy and the Postmodern Turn.” Critical Literacy: Policy, Praxis, and the Postmodern. Eds. Albany, NY: SUNY Press

Leedy, P.D., & Ormrod, J.E. (2005). Practical Research Planning and Design (Eighth Edition). Upper Saddle River, NJ: Pearson Education Inc.

McLaren, P. L. (1991). “Field Relations and the Discourse of the Other: Collaboration in Our Own Ruin.” Experiencing Fieldwork: An Inside View of Qualitative Research. Shaffir, W. B., & Stebbins, R. A., Eds. Newbury Park, CA: Sage

Meier, D., Kohn, A., Darling-Hammond, L., Sizer, T.R., Wood, G. (2004). Many Children Left Behind. Boston, MA: Beacon Press

Meyer, E., Abrami, P.C., Wade, C.A., Aslan, O., & Deault, L. (2010). Improving Literacy and Metacognition with Electronic Portfolios: Teaching and Learning with ePearl. Computers & Education, 55, 84-91. doi:10.1016/j.compedu.2009.12.005

Mills, E.S., Hamilton, B.W. (1994). Urban Economics. New York, NY: HarperCollins College Publishers

Nielsen J. (1993). Usability Engineering. Boston, MA: Academic.

Ocak, G., & Ulu, M. (2009). The Views of Students, Teachers, and Parents and the Use of Portfolio at the Primary Level. Procedia Social and Behavioral Sciences, 1, 28-36. doi:10.1016/j.sbspro.2009.01.009

Peacock, S., Gordon, L., Murray, S., Morss, K., & Dunlop, G. (2010). Tutor Response to Implementing an ePortfolio to Support Learning and Personal Development in Further and Higher Education Institutions in Scotland. British Journal of Educational Technology, 41.5, 827-851. doi:10.1111/j.1467-8535.2009.00986.x

Publication Manual of the American Psychological Association. (Sixth Edition). (2010). Washington, DC: American Psychological Association

Schneider, C.G. (2008). The Proof Is in the Portfolio. Liberal Education, 95.1. Retrieved from http://www.aacu.org/liberaleducation/le-wi09/le-wi09_president.cfm

Scowen, G., Regenbrecht, H. (2009). Increased Popularity Through Compliance With Usability Guidelines in E-Learning Web Sites. International Journal of Information Technology and Web Engineering, 4.3, 38-57. doi:10.4018/jitwe.2009100603

Seitz, H, & Bartholomew, C. (2008). Powerful Portfolios for Young Children. Early Childhood Education Journal, 36, 63-68. doi:10.1007/s10643-008-0242-7

Seybolt, R.F. (1935). The Public Schools of Colonial Boston, 1635-1775. Boston, MA: Harvard University Press

Sing, C.C., Der-Thanq, V.C. (2004). A Review of Usability Evaluation Methods for Instructional Multimedia: An Analytical Framework. International Journal of Instructional Media, 31.3, 229-238. Retrieved from http://www.emich.edu.ezproxy.emich.edu/library

Skov, M.B., Stage, J. (2009). A Conceptual Tool for Usability Problem Identification in Website Development. International Journal of Information Technology and Web Engineering, 4.4, 22-35. doi:10.4018/jitwe.2009100102

Teoh, K.K., Ong, T.S., Lim, P.W., Liong, R.P.Y., Yap, C.Y. (2009). Explorations of Web Usability. American Journal of Applied Sciences, 6.3, 424-429. Retrieved from http://www.emich.edu.ezproxy.emich.edu/library

Vermilion, E.R. (2008). The EdVantage of Efolios. Distance Learning, 5.4, 67-72. Retrieved from http://www.infoagepub.com/distance-learning.html

Wall, K., Higgins, S., Miller, J., & Packard, N. (2006). Developing Digital Portfolios: Investigating How Digital Portfolios Can Facilitate Pupil Talk About Learning. Technology, Pedagogy and Education, 15.3, 261-273. doi:10.1080/14759390600923535

Zaharias, P. (2009). Usability in the Context of e-Learning: A Framework Augmenting ‘Traditional’ Usability Constructs with Instructional Design and Motivation to Learn. International Journal of Technology and Human Interaction, 5.4, 37-59. doi:10.4018/jthi.20090

Advertisements

Create a free website or blog at WordPress.com.