Educational Scholarship in the Digital Age: A Scoping Review and Analysis of Scholarly Products

  • Brent Thoma 1 2 3
  • Teresa Chan 1 4
  • Javier Benitez 1
  • Michelle Lin 1 5
  1. 1.  MedEdLIFE Research Collaborative
  2. 2.  Emergency Medicine Residency Program, University of Saskatchewan
  3. 3.  Simulation Fellowship Program, Massachusetts General Hospital
  4. 4.  Department of Medicine, Division of Emergency Medicine, McMaster University
  5. 5.  Department of Emergency Medicine, University of California San Francisco


Boyer’s framework of scholarship was published before significant growth in digital technology. As more digital products are produced by medical educators, determining their scholarly value is of increasing importance. This scoping systematic review developed a taxonomy of digital products and determined their fit within Boyer’s framework of scholarship. We conducted a broad literature search for descriptions of digital products in the medical literature in July 2013 using Medline, EMBASE, ERIC, PSYCHinfo, and Google Scholar. A framework analysis categorized each product using Boyer’s model of scholarship, while a thematic analysis defined a taxonomy of digital products. 7422 abstracts were found and 524 met inclusion criteria. Digital products mapped primarily to the scholarship of teaching (85.4%) followed by integration (7.6%), application (5.5%), and discovery (1.5%). A taxonomy of 19 categories was defined. Web-based or computer assisted learning (41%) was described most frequently. We found that digital products are well described in medical literature and fit into Boyer’s framework of scholarship and proposed a taxonomy of digital products that parallel traditional forms of the scholarship of teaching and learning. This research should inform the development of tools to examine the impact and quality of digital products.


In 1990 Boyer redefined the scope of scholarship in higher education with the definition of four overlapping subtypes of scholarship (discovery, integration, application, and teaching) (Boyer 1990). Prior to this redefinition, scholarship was largely considered to consist only of the discovery subtype. Boyer’s influential definition paved the way for the recognition of a broader definition of scholarship that included teaching in addition to research. The explosive growth of digital products (resources used for the dissemination of information that exist primarily in digital formats) that has occurred since the internet was democratized in 1995 could not be predicted at that time (Leiner et al. 2009). Social media, online courses, blogs, podcasts and other digital products have since changed the way we teach, disseminate, and discuss scholarly ideas. Their exclusion from traditional scholarly frameworks, combined with a lack of standards to ensure their quality, may explain why they are generally not viewed as scholarship by members of the academic establishment (Brabazon 2006; Hendricks 2010; Kirkup 2010; Savage 2006).

Scholars and educators are turning to digital methods for disseminating knowledge and reaching students (Priem 2013). This has resulted in the creation of online communities of practice with benefits including: increased collaboration, enhanced knowledge dissemination, instantaneous scholarly discussion, and the generation of scholarly identity (Kirkup 2010; Gruzd, Staves, and Wilk 2010; Maitzen 2012; Shema, Bar-Ilan, and Thelwall 2012). Arguments against digital products note that they have not proven to be superior and that they require more time to develop (Cooke 2014). The increasing prominence of digital products in medical education and the time being devoted to their development makes determining their scholarly value extremely important (Cadogan et al. 2014; Matava et al. 2013; Bahner et al. 2012).

In this scoping review paper, we quantify the increasing prevalence of digital products in the medical literature, develop a taxonomy of digital products, and compare the products in the taxonomy to traditional forms of the scholarship of teaching and learning. We hope that this will increase the awareness of this growing area of educational scholarship and classify digital products so that their value can be understood within the context of their traditional parallels.



In concert with an expert librarian, an expert search strategy was developed using the Medline, EMBASE, ERIC, and PSYCHinfo databases, as they were deemed to be the most likely to provide literature on digital products used in medical education. The search was not limited by year or language, and used the keywords and keyword variations of: (student, medical OR medical student OR “internship and residency” OR intern OR resident) AND (education, medical OR education, medical, graduate OR education, medical, undergraduate OR “medical education”) AND (blog OR weblog OR microblog OR social media OR social network OR “health 2.0” OR “web 2.0” OR video OR youtube OR podcast OR vodcast OR webcast OR screencast OR wiki OR widget OR new media OR new technology OR mobile app OR app, collaborative OR cooperative behavior OR conferencing OR crowdsource OR RSS OR “really simple syndication” OR computer-assisted instruction OR web-based instruction OR “access to information” OR open access OR free access).

In addition to this traditional literature search, a previously described Google Scholar search methodology (Chan et al. 2012) was conducted for five sets of keywords: “blogging and scholarship,” “digital scholarship medicine medical,” “free open access medical education,” “medical blogging” and “’tenure and promotion blogging.” The first 500 results for each keyword set were reviewed and relevant results were added to the findings.

A title review of the abstracts was performed by one author (BT). Abstracts were excluded if (1) there was no English-language abstract, (2) they were duplicates, or (3) they clearly did not address the use of digital products in medicine. The abstracts were coded and classified with a detailed abstract review conducted by two authors (BT, JB). Upon abstract review, articles were excluded if (1) no particular digital product was described, (2) the digital product did not meet the criteria for scholarship based on Boyer’s model, or (3) upon closer inspection they met the initial exclusion criteria.

During the abstract review, two authors (BT, JB) performed both a framework analysis and thematic analysis of the digital products described in the abstracts. Two reviewers (BT, JB) classified the digital products described in the first 60 abstracts collaboratively to develop an initial taxonomy and set of definitions for the thematic analysis and to calibrate the coding schemes for the thematic and framework analyses. Subsequently a constant comparator technique was used to perform both analyses whereby classifications were made independently in batches of approximately 100 abstracts and compared. The frequent comparisons allowed the reviewers to ensure consistency within the analyses and to refine a consensus definition for each type of digital product in the thematic analysis.

When available and necessary, full manuscripts were reviewed to accurately classify the digital products and their form of scholarship. Discordant classifications were discussed by the reviewers and resolved by consensus when possible. When consensus was not reached, a third reviewer (TC) arbitrated disagreements. The third reviewer also audited the excluded abstracts to ensure that they met the review’s exclusion criteria. The year of publication of each abstract was also recorded to demonstrate the prevalence of digital products described each year.

While they were conducted concurrently, the two analyses were functionally independent. The thematic analysis was used to derive a taxonomy that defined the described all of the digital products found in the literature. Additional items were added to the taxonomy as they were found and the definitions were frequently refined to accurately describe all of the digital products effectively.

The purpose of the framework analysis was to determine if and how digital products fit into Boyer’s four types of scholarship (Boyer, 1990). Digital products were classified as one or more of Boyer’s types of scholarship: discovery (original research for the advancement of knowledge), integration (contextualizing information across disciplines or into larger intellectual patterns), application (applying knowledge dynamically to inform and test new theories in an engaged fashion), and/or teaching (systematic study of teaching and learning in the presence of learners) (Gale et al. 2013; Boyer 1990). The intraclass correlation coefficient was calculated to determine a measure of agreement.

The definitions resulting from the thematic analysis were assessed to determine if there were traditional scholarly products used for the same purpose. This comparison, while inherently subjective, was conducted to further contextualize the role of each type of digital product.



The flow diagram for the literature search, title review, and abstract review is presented in Figure 1. The thematic and framework analyses were conducted on digital products described by the 524 abstracts that met the inclusion criteria. An abstract published in 1974 described the oldest digital product.


Figure 1. Diagram illustrating the number of articles excluded through the title and abstract reviews.

The number of digital products described in the published medical literature between 1974 and July 2013 is illustrated in Figure 2. The number of digital products for 2013 was projected to double because our literature search only included articles published through July 2013.



Description: Macintosh HD:Users:TChanMD:Dropbox:Social Media and Scholarship (1):The Winnower:Figure 2.001.jpg

Figure 2. The number of digital products described in the medical literature over time.


Framework Analysis

Table 1 presents the results of the analysis mapping published digital products to Boyer’s framework of scholarship.(Boyer 1990) The intraclass correlation between the raters was 0.65, but disagreements were ultimately discussed to resolve consensus. Most products (85.4%) were categorized under the scholarship of teaching. The scholarship of integration (7.6%), application (5.5%), and discovery (1.5%) were described much less frequently. This table further stratifies these scholarship models based on the 19 categories of digital products, as derived by our thematic analysis. Of note, there were some products that could be classified as more than one type of scholarship.


Table 1: Types and numbers of digital products mentioned in the literature and classified using Boyer's Framework of Scholarship

Digital Product






Web-based or computer assisted learning

0 (0)

1 (0.5)

2 (0.9)

214 (98.6)*


Multi-modal products

0 (0)

7 (14.0)

3 (6.0)

40 (80.0)*


Social network

0 (0)

13 (38.2)*

8 (23.5)

13 (38.2)*


Instructional video

0 (0)

0 (0)

2 (6.5)

29 (93.5)


Online repository

0 (0)

8 (27.6)

4 (13.8)

17 (58.6)*



0 (0)

0 (0)

0 (0)

28 (100)


Online course

0 (0)

0 (0)

0 (0)

27 (100)*


Video podcast

0 (0)

0 (0)

2 (7.5)

25 (92.6)



0 (0)

4 (15.4)

3 (11.5)

19 (73.1)*


Open access journal

8 (50)*

7 (43.8)

0 (0)

1 (6.3)



0 (0)

0 (0)

0 (0)

16 (100)*



0 (0)

1 (6.7)

2 (13.3)

12 (80)*


Online discussion board

0 (0)

0 (0)

0 (0)

8 (100)*



0 (0)

0 (0)

0 (0)

7 (100)*


Application ("app")

0 (0)

0 (0)

3 (75)

1 (25)


Online textbook

0 (0)

0 (0)

0 (0)

2 (100)*


Virtual reality

0 (0)

0 (0)

0 (0)

2 (100)*


Search engine

0 (0)

0 (0)

1 (100)

0 (0)


Serious game

0 (0)

0 (0)

0 (0)

1 (100)*



8 (1.5)

41 (7.6)

30 (5.5)

462 (85.4)


In table 1, the starred numbers represent the most popular type of scholarship for each product. The table includes 17 abstracts that were classified as multiple forms of scholarship, resulting in totals (541) greater than the number of abstracts reviewed (524).


Thematic Analysis

Table 2 provides a taxonomy of the digital products described in the literature and derived from the thematic analysis. Each of the 19 categories are defined with an example provided. Together, web-based learning and computer assisted learning (41%) were the most prevalent forms of digital product. A single category was created for these two types of digital products because prior to the democratization and widespread accessibility of the internet, web-based learning products were classified under the umbrella term of computer assisted learning. The significant overlap between these two terms necessitated their amalgamation into one category in our taxonomy. Social networks, instructional videos, online repositories, podcasts, online courses, video podcasts (also known as screencasts or vodcasts), and blogs had roughly similar prevalence and collectively comprised another 37% of the publications.


Table 2: Definitions and examples of digital products.

Digital Product



Applications (‘apps’)

A resource downloaded to a smartphone.

iRash is an application that allows users to search and learn about various rashes (Deveau and Chilukuri 2012)


A website used to publish information in periodic posts that are primarily text-based.

A blog was created to host synopses of ‘morning report’ sessions run by chief medical residents (Bogoch et al. 2012)


A common form of direct electronic messaging between a sender and one or more recipients.

E-mail was used to send questions to teach residents about pediatric emergency medicine (Komoroski 1998)

Instructional Video

A video demonstrating a skill (ie procedure, physical exam finding, ECG or x-ray interpretation, etc). 

Instructional video used to teach chest tube insertion (Davis et al. 2012)

Multi-modal products

A product that consists of multiple digital products.

An online course on evidence based medicine and critical appraisal that used video podcasts, a wiki and blogs (Tam and Eastwood 2012)

Online Course

A complete curriculum delivered using multiple online modalities. Differs from multi-modal products in that it is organized into a formal curriculum.

The Online Genetic Testing Curriculum is a course about the ethical, legal, and social implications of genetic testing and counseling (Metcalf, Tanner, and Buchanan 2010)

Online Discussion Board

An online forum that allows users to post and respond to other participants.

A clinical discussion board for learners to describe their rural medicine experiences (Baker, Eley, and Lasserre 2005)

Online Repository

An online database that resources can be drawn from and added to.

A repository of images of dermatologic findings in darker-skinned patients (Ezzedine et al. 2008)

Online Textbook

A textbook published online.

ODITEB1 (Open Distributed Text Book), an online textbook that describes the diagnosis of gastrointestinal tumours (Horsch et al. 1998)

Open Access Journal

A journal only available online that publishes articles without access restrictions.

Various online journals have been created to decrease cost and allow open-access publication of scientific materials (Davis and Walters 2011)


Audio recordings that are published periodically with the intent of disseminating knowledge.

Surgery 101 podcasts are used to teach core principles to clinical clerks on their surgical rotation (White, Sharma, and Boora 2011)

Search Engine

Search engines used to find information online.

Google, Yahoo, Dogpile, Altavista, Metacrawlers and Ask were used to find information on scleroderma renal crisis (Akbar and Yacyshyn 2009)

Serious Game

An online game designed to educate the players.  

eMedOffice, a serious game to teach practice management.(Hannig et al. 2012)

Social Network

An online platform that allows synchronous and asynchronous communication between individuals.

Twitter used to connect teachers with learners (Forgie, Duff, and Ross 2013)

Video Podcast

Videos with embedded audio that are published periodically. Differs from instructional videos because it focuses on knowledge rather than skill.

Video podcasts used to teach embryology (Evans 2011)

Virtual Reality

A virtual environment used to present learning material.

A virtual reality simulator was used to simulate medical cases (Alverson et al. 2008)

Web Based Learning or Computer Assisted Learning

Educational modules that may make use of multiple modalities. Web-based learning is based online while computer assisted learning is not. These modalities were combined due to substantial overlap.

A web based module on pediatric pain management (Ameringer et al. 2012)

A computer based application about occupational lung disease (Bresnitz, Gracely, and Rubenstein 1992)


An online webpage that cannot be classified as any other digital product.

Case Based Pediatrics is a website with a list of teaching cases for medical students and residents (Falagas, Karveli, and Panos 2007)


A website that can be openly edited by end-users. Utilizes crowd-sourcing as a method for improving and revising the content.

A wiki site for orthopedic cases, utilizes a scoreboard to encourage participation (Ma et al. 2008)


Historical Parallels

As demonstrated by our framework analysis, digital products can be classified within the types of scholarship described by Boyer (Boyer 1990) and most fall under teaching and learning. Following the completion of our thematic analysis, the definitions of the digital products were compared with traditional forms of the scholarship of teaching and learning. Table 3 outlines the parallels between traditional products and 18 of the 19 digital products described in the thematic analysis. No product was found that was comparable to the digital product ‘virtual reality.’

Table 3: Comparing traditional products used for the scholarship of teaching and learning to digital products that are used for this purpose

Types of teaching and learning resources

Examples of
Traditional Products

Examples of
Digital Products

Interactive resources

Small groups

Online discussion board
Social network

Independent study resources

Discussions with tutors
Group work
Laboratory work

Online course
Serious game
Virtual reality
Web based and Computer assisted learning

Audiovisual resources

Skill demonstration

Video podcast
Instructional video

Point-of-care resources


Applications (‘apps’)


Written resources

Printed journals
Medical journalism

Online textbook
Open access journal

Resource repository

Library classification system

Online repository
Search engine



The growing number of digital products documented in the literature (Figure 1 and 2) suggests that medical educators are increasingly using technology to engage in various forms of scholarship. While educators have discussed applying Boyer’s traditional definitions of scholarship to digital products (Heap and Minocha 2012; Pearce et al. 2010), we provide the first comprehensive framework analysis of these products.

Our framework analysis found that, following teaching and learning, integration (7.6%), application (5.5%), and discovery (1.5%) were the most frequent types of scholarship found in digital products. We suspect that the digital products were predominantly consistent with scholarship of teaching and learning because, despite Boyer’s reclassification of scholarship, educators have traditionally not had their scholarly contributions recognized. Literature that assesses their innovations is one way to receive academic recognition for their work. Educators should keep in mind that digital products can be scholarly outside of their traditional realm of teaching. For example, Boyer’s concept of application was demonstrated by the various ‘apps’ that allow translation of concepts at the point of care (Graber, Tompkins, and Holland 2009), integration was illustrated by an online textbook that synthesized multiple resources into a single resource (Horsch et al. 1998), and discovery was exemplified by open access online journals that fostered new scientific works (P. M. Davis and Walters 2011). Social networks were the most versatile product with multiple examples of their use in teaching, application, and integration.

The thematic analysis described the diversity of digital products (Table 2). Notably, web-based and computer assisted learning programs were prominently featured in the literature and there has been a recent uptake of social media (Nickson and Cadogan 2014; Cadogan et al. 2014). Social networks, in particular, seem to have impacted medical education by allowing scholars to share their digital products (Boulos, Maramba, and Wheeler 2006).

A traditional parallel was found for nearly every digital product defined in the thematic analysis. The use of digital products was particularly prominent for the scholarship of teaching and learning. This may be because of their reach, customization, and updatability. Whereas scholarly teaching was historically a fleeting event offered to a defined group (i.e. an address that was given in a lecture hall), digital products extend their reach to large numbers of learners who can access them at their convenience. This asynchrony allows learners to customize their experience (i.e. by speeding up or slowing down a lecture) and educators to update their products as needed.

That said, there is no compelling evidence that digital products are more effective for learning and they may take more time and resources to develop than traditional products (Cooke 2014). They have also been criticized for their lack of editorial oversight and review (Brabazon 2006; Kirkup 2010). These limitations may limit their widespread endorsement and utilization. Further research will be required to determine when and how they should be used.

While our results suggest that this research is increasingly being conducted, the role and value of digital products in our current academic schema for scholarship remains poorly defined, and hence, poorly acknowledged. Institutions that do acknowledge digital products as scholarship for the purpose of promotion and tenure decisions have difficulty classifying them and quantifying their value relative to other scholarly pursuits (Gruzd, Staves, and Wilk 2010; Cheverie, Boettcher, and Buschman 2009; Rockwell 2011; Ruiz, Mintzer, and Leipzig 2006). Novel ways to recognize digital products include publishing them on a platform with peer review and publication processes such as MedEdPORTAL (Ruiz, Mintzer, and Leipzig 2006; Reynolds and Candler, Christopher 2008) or conducting educational research to evaluate their efficacy (Cheston, Flickinger, and Chisolm 2013). Regardless, the amount of academic recognition for digital products is relatively low compared to the effort expended to build and maintain them and may limit their growth in the future (Anderson et al. 2013; ProfHacker 2012).



While our literature search was intended to be as broad as possible, it is still likely that some digital products were missed since they may not have been reported in the literature. A broader review of grey and non-English literature would not have been feasible given the sheer volume of unreported products. For example, a recent report found that there were 183 English-language blogs and podcasts in emergency medicine alone (Cadogan et al. 2014). Additionally, we may have missed digital products of historic significance that were described using terms that are not applicable today. For example, CD-ROM’s were likely to have been considered digital products in the past but were not included in our literature search. Missing resources would change the number of products per year represented in Figure 2 and made our taxonomy of digital products incomplete.

The exclusion of the MedEdPORTAL database could also be considered a limitation as it publishes many digital products. However, our search explicitly attempted to quantify and describe the digital products described in the literature. MedEdPORTAL’s publications are digital products, rather than descriptions of them, and for this reason they were considered to be outside of the scope of this review.

Finally, our quantification of the rapidly increasing number of digital products described annually in the literature fails to account for the increase in literature that has been published in general (Larsen, 2010). Unfortunately, we were unable accurately quantify this growth for the body of literature that our review assessed. As the amount of research published annually is increasing (Larsen, 2010), the increase in descriptions of digital products would have been less spectacular had we been able to take this into account.


Future Directions

Since the digital products described in the medical literature fit within Boyer’s framework, we feel strongly that they should be considered alongside other forms of scholarship.  However, given the ease with which some products can be created, better evaluation tools will need to be developed to determine their quality, value, and relative impact. Educator portfolios are becoming accepted as a way to provide additional detail to the traditional curriculum vitae, which sub-optimally captures the scholarly efforts of educators (Simpson et al. 2007; Baldwin, Chandran, and Gusic 2011).

In showing that digital products fall within Boyer’s framework of scholarship, our findings suggest that we should look to apply other conceptual frameworks of educational scholarship to digital products or online educational resources. Frequently, educators lean towards the criteria for assessing scholarship developed by Glassick. Assessment frameworks such as Glassick’s criteria of scholarship are manifest in the AAMC Toolbox for Evaluating Educators and could be used to evaluate these portfolios (Glassick 2000; Gusic et al. 2013). Table 3 suggests multiple parallels between traditional and digital projects for teaching and learning that could guide how digital products should fit into these portfolios. Developing a standardized approach would allow promotion committees and administrative leadership to evaluate digital and traditional educational efforts more rigorously.

Together, Boyer and Glassick’s respective frameworks provide a roadmap for educators interested in scholarship. Digital scholars must take care to ensure that their digital products warrant scholarly respect by ensuring that they stand up to the scrutiny of these recognized conceptual frameworks.



Digital products are increasingly being described in the medical literature. They are likely to have a substantial impact on medical education and can readily fit into Boyer’s established framework of scholarship. Our taxonomy shows clear parallels between digital and traditional products and can hopefully provide a framework for further research on digital scholarship.




Akbar, S, and E Yacyshyn. 2009. “Is There Relevant Information about Scleroderma Renal Crisis on Most Frequently Visited Internet Search Engines?” Journal of Rheumatology 36 (11): 2563–64.

Alverson, Dale C, Stanley M Saiki, Summers Kalishman, Marlene Lindberg, Stewart Mennin, Jan Mines, Lisa Serna, et al. 2008. “Medical Students Learn over Distance Using Virtual Reality Simulation.” Simulation in Healthcare : Journal of the Society for Simulation in Healthcare 3 (1): 10–15. doi:10.1097/SIH.0b013e31815f0d51.

Ameringer, Suzanne, Deborah Fisher, Sue Sreedhar, Jessica M Ketchum, and Leanne Yanni. 2012. “Pediatric Pain Management Education in Medical Students: Impact of a Web-Based Module.” Journal of Palliative Medicine 15 (9): 978–83. doi:10.1089/jpm.2011.0536.

Anderson, Michael G, Donna D Alessandro, Dawn Quelle, Rick Axelson, Lois J Geist, and Donald W Black. 2013. “Recognizing Diverse Forms of Scholarship in the Modern Medical College”, 120–25. doi:0.5116/ijme.51b4.730c.

Bahner, David P, Eric Adkins, Nilesh Patel, Chad Donley, Rollin Nagel, and Nicholas E Kman. 2012. “How We Use Social Media to Supplement a Novel Curriculum in Medical Education.” Medical Teacher 34 (6): 439–44. doi:10.3109/0142159X.2012.668245.

Baker, Peter G, Diann S Eley, and Kaye E Lasserre. 2005. “Tradition and Technology: Teaching Rural Medicine Using an Internet Discussion Board.” Rural and Remote Health 5 (4): 435.

Baldwin, Constance, Latha Chandran, and Maryellen Gusic. 2011. “Guidelines for Evaluating the Educational Performance of Medical School Faculty: Priming a National Conversation.” Teaching and Learning in Medicine 23 (3): 285–97. doi:10.1080/10401334.2011.586936.

Bogoch, Isaac I, David W Frost, Suzanne Bridge, Todd C Lee, Wayne L Gold, Daniel M Panisko, and Rodrigo B Cavalcanti. 2012. “Morning Report Blog: A Web-Based Tool to Enhance Case-Based Learning.” Teaching and Learning in Medicine 24 (3): 238–41. doi:10.1080/10401334.2012.692273.

Boulos, Maged N Kamel, Inocencio Maramba, and Steve Wheeler. 2006. “Wikis, Blogs and Podcasts: A New Generation of Web-Based Tools for Virtual Collaborative Clinical Practice and Education.” BMC Medical Education 6 (January): 41. doi:10.1186/1472-6920-6-41.

Boyer, E. 1990. “Scholarship Reconsidered: Priorities of the Professoriate” The Carnegie Foundation for the Advancement of Teaching: Princeton, NJ.

Brabazon, T. (2006). The Google effect: Googling, blogging, wikis and the flattening of expertise. Libri, 56(3), 157-167. doi:10.1515/LIBR.2006.157

Bresnitz, Eddy A, Edward J Gracely, and Harriet L Rubenstein. 1992. “A Randomized Trial to Evaluate a Computer-Based Learning Program in Occupational Lung Disease.” Journal of Occupational and Environmental Medicine 34 (4).

Cadogan, M., B. Thoma, T. M. Chan, and M. Lin. 2014. “Free Open Access Meducation (FOAM): The Rise of Emergency Medicine and Critical Care Blogs and Podcasts (2002-2013).” Emergency Medicine Journal, (February). doi:10.1136/emermed-2013-203502.

Chan, Teresa M, Clare Wallner, Thomas K Swoboda, Katrina a Leone, and Chad Kessler. 2012. “Assessing Interpersonal and Communication Skills in Emergency Medicine.” Academic Emergency Medicine 19 (12): 1390–1402. doi:10.1111/acem.12030.

Cheston, Christine C, Tabor E Flickinger, and Margaret S Chisolm. 2013. “Social Media Use in Medical Education: A Systematic Review.” Academic Medicine : Journal of the Association of American Medical Colleges 88 (6): 893–901. doi:10.1097/ACM.0b013e31828ffc23.

Cheverie, Joan F., Jennifer Boettcher, and John Buschman. 2009. “Digital Scholarship in the University Tenure and Promotion Process: A Report on the Sixth Scholarly Communication Symposium at Georgetown University Library.” Journal of Scholarly Publishing 40 (3): 219–30. doi:10.1353/scp.0.0044.

Cooke, David. 2014. “Futurecasting in education technologies: Fun new toys and a reality check.” International Conference on Residency Education, Plenary Session. Available at: Retrieved November 25, 2014.

Davis, James S, George D Garcia, Mary M Wyckoff, Salman Alsafran, Jill M Graygo, Kelly F Withum, and Carl I Schulman. 2012. “Use of Mobile Learning Module Improves Skills in Chest Tube Insertion.” The Journal of Surgical Research 177 (1). Elsevier Ltd: 21–26. doi:10.1016/j.jss.2012.03.022.

Davis, Philip M, and William H Walters. 2011. “The Impact of Free Access to the Scientific Literature: A Review of Recent Research.” Journal of the Medical Library Association : JMLA 99 (3): 208–17. doi:10.3163/1536-5050.99.3.008.

Deveau, Michael, and Suneel Chilukuri. 2012. “Mobile Applications for Dermatology.” Seminars in Cutaneous Medicine and Surgery 31 (3). Elsevier Inc. 174–82. doi:10.1016/j.sder.2012.06.001.

Evans, Darrell J R. 2011. “Using Embryology Screencasts: A Useful Addition to the Student Learning Experience?” Anatomical Sciences Education 4 (2). Wiley Subscription Services, Inc., A Wiley Company: 57–63. doi:10.1002/ase.209.

Ezzedine, K, a Amiel, P Vereecken, T Simonart, B Schietse, K Seymons, B S Ndiaye, et al. 2008. “Black Skin Dermatology Online, from the Project to the Website: A Needed Collaboration between North and South.” Journal of the European Academy of Dermatology and Venereology : JEADV 22 (10): 1193–99. doi:10.1111/j.1468-3083.2008.02762.x.

Falagas, Matthew E, Efthymia a Karveli, and George Panos. 2007. “Infectious Disease Cases for Educational Purposes: Open-Access Resources on the Internet.” Clinical Infectious Diseases : An Official Publication of the Infectious Diseases Society of America 45 (4): 495–500. doi:10.1086/519977.

Forgie, Sarah Edith, Jon P Duff, and Shelley Ross. 2013. “Twelve Tips for Using Twitter as a Learning Tool in Medical Education.” Medical Teacher 35 (1): 8–14. doi:10.3109/0142159X.2012.746448.

Gale, Nicola K, Gemma Heath, Elaine Cameron, Sabina Rashid, and Sabi Redwood. 2013. “Using the Framework Method for the Analysis of Qualitative Data in Multi-Disciplinary Health Research.” BMC Medical Research Methodology 13 (1). BMC Medical Research Methodology: 117. doi:10.1186/1471-2288-13-117.

Glassick, Charles E. 2000. “Boyer’s Expanded Definitions of Scholarship, the Standards for Assessing Scholarship, and the Elusiveness of the Scholarship of Teaching.” Academic Medicine 75(9), 877–80. doi:10.1097/00001888-200009000-00007

Graber, Mark L, David Tompkins, and Joanne J Holland. 2009. “Resources Medical Students Use to Derive a Differential Diagnosis.” Medical Teacher 31: 522–27. doi:10.1080/01421590802167436.

Gruzd, Anatoliy, Kathleen Staves, and Amanda Wilk. 2010. “Tenure and Promotion in the Age of Online Social Media.” Proceedings of the American Society for Information Science and Technology 48 (1): 1–9. doi:10.1002/meet.2011.14504801154.

Gusic, M, J Amiel, C Baldwin, L Chandran, R Fincher, B Mavis, P O’Sullivan, et al. 2013. “Using the AAMC Toolbox for Evaluating Educators: You Be the Judge!” MedEdPORTAL. doi:10.15766/​mep_2374-8265.9313.

Hannig, Andreas, Nicole Kuth, Monika Özman, Stephan Jonas, and Cord Spreckelsen. 2012. “eMedOffice: A Web-Based Collaborative Serious Game for Teaching Optimal Design of a Medical Practice.” BMC Medical Education 12 (January): 104. doi:10.1186/1472-6920-12-104.

Heap, Tania, and Shailey Minocha. 2012. “An Empirically Grounded Framework to Guide Blogging for Digital Scholarship.” Research in Learning Technology 20 (August). doi: 10.3402/rlt.v20i0.19195.

Hendricks, Arthur. 2010. “Bloggership, or Is Publishing a Blog Scholarship? A Survey of Academic Librarians.” Library Hi Tech 28 (3): 470–77. doi:10.1108/07378831011076701.

Horsch, A., P. Hellerhoff, M. Hogg, H. Ahlbrink, T. Balbacha, Liss. T., K. Minov, and P. Gerhardt. 1998. “Concepts of a Web-Based Open Distributed Textbook for the Multimodal Diagnostics of Gastrointestinal Tumours with MRI, CT and Video-Endoscopy Addressing Students of Medicine and Students of Medical Informatics as Two Different Target Groups.” Studies in Health Technology and Informatics 52 (2): 793–97.

Kirkup, Gill. 2010. “Academic Blogging: Academic Practice and Academic Identity.” London Review of Education 8 (1): 75–84. doi:10.1080/14748460903557803.

Komoroski, E M. 1998. “Use of E-Mail to Teach Residents Pediatric Emergency Medicine.” Archives of Pediatrics & Adolescent Medicine 152 (11): 1141–46. doi:10.1001/archpedi.152.11.1141  

Larsen, P. O., & von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics, 84(3), 575-603. doi:10.1007/s11192-010-0202-z.

Leiner, Barry M, David D Clark, Robert E Kahn, Leonard Kleinrock, Daniel C Lynch, Jon Postel, Larry G Roberts, and Stephen Wolff. 2009. “A Brief History of the Internet.” Computer Communication Review 39 (5): 22–31. doi:10.1145/1629607.1629613

Ma, Zhen-Sheng, Hong-Ju Zhang, Tao Yu, Gang Ren, Guo-Sheng Du, and Yong-Hua Wang. 2008. “ Case-Based Orthopaedic Wiki Project in China.” Clinical Orthopaedics and Related Research 466 (10): 2428–37. doi:10.1007/s11999-008-0396-z.

Maitzen, Rohan. 2012. “Scholarship 2.0: Blogging And/as Academic Practice.” Journal of Victorian Culture 17 (3): 348–54. doi:10.1080/13555502.2012.689502.

Matava, Clyde T, Derek Rosen, Eric Siu, and Dylan M Bould. 2013. “eLearning among Canadian Anesthesia Residents: A Survey of Podcast Use and Content Needs.” BMC Medical Education 13 (January): 59. doi:10.1186/1472-6920-13-59.

Metcalf, Mary P, T Bradley Tanner, and Amanda Buchanan. 2010. “Effectiveness of an Online Curriculum for Medical Students on Genetics, Genetic Testing and Counseling.” Medical Education Online 15 (January): 1–12. doi:10.3402/meo.v15i0.4856.

Nickson, Christopher P, and Michael D Cadogan. 2014. “Free Open Access Medical Education (FOAM) for the Emergency Physician.” Emergency Medicine Australasia 26 (1): 76–83. doi:10.1111/1742-6723.12191.

Pearce, Nick, Martin Weller, Eileen Scanlon, and Melanie Ashleigh. 2010. “Digital Scholarship Considered: How New Technologies Could Transform Academic Work.” In Education 16 (1).

Priem, Jason. 2013. “Beyond the Paper.” Nature 495: 437–40. doi:10.1038/495437a.

Koh, Adeline. 2012. “The Challenges of Digital Scholarship.” Chronicle of Higher Education. Retrieved from on D\necember 10, 2014.

Reynolds, Robby J., and S. Candler, Christopher. 2008. “MedEdPORTAL : Educational Scholarship for Teaching.” Journal of Continuing Education in the Health Professions 28 (2): 91–94. doi:10.1002/chp.

Rockwell, Geoffrey. 2011. “On the Evaluation of Digital Media as Scholarship.” Profession 2011 1: 152–68. doi:10.1632/prof.2011.2011.1.152.

Ruiz, Jorge G, Michael J Mintzer, and Rosanne M Leipzig. 2006. “The Impact of E-Learning in Medical Education.” Academic Medicine 81 (3): 207–12. doi:10.1097/00001888-200603000-00002

Savage, William W. 2006. “The Transom: You Can’t Spill Mustard on a Blog.” Journal of Scholarly Publishing 38 (1): 47–51. doi:10.1353/scp.2007.0005.

Shema, Hadas, Judit Bar-Ilan, and Mike Thelwall. 2012. “Research Blogs and the Discussion of Scholarly Information.” PloS One 7 (5): e35869. doi:10.1371/journal.pone.0035869.

Simpson, Deborah, Ruth-Marie E Fincher, Janet P Hafler, David M Irby, Boyd F Richards, Gary C Rosenfeld, and Thomas R Viggiano. 2007. “Advancing Educators and Education by Defining the Components and Evidence Associated with Educational Scholarship.” Medical Education 41 (10): 1002–9. doi:10.1111/j.1365-2923.2007.02844.x.

Tam, Chun Wah Michael, and Anne Eastwood. 2012. “Available, Intuitive and Free! Building E-Learning Modules Using Web 2.0 Services.” Medical Teacher 34 (12): 1078–80. doi:10.3109/0142159X.2012.731105.

White, J S, N Sharma, and P Boora. 2011. “Surgery 101: Evaluating the Use of Podcasting in a General Surgery Clerkship.” Medical Teacher 33 (11): 941–43. doi:10.3109/0142159X.2011.588975.


Showing 8 Reviews

  • Img 20150909 170817 edit edit
    Allan McDougall
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    Just a few terminology questions for you to consider:
    Could you provide a definition of Digital Product up front? And perhaps Digital Scholarship as well? I'm wondering if it's sufficient to say that Scholarship involving Digital Products is Digital Scholarship? Does this create a dichotomy between Digital and Non-Digital Scholarship? 

    Structure & Scope

    This is a well-structured paper that does a great job handling a question with a very broad scope. The authors have done a lot of work here and it is presented effectively and clearly. 

    The article is well-written.
    2 minor points:
    Why on page 6 is there extra space between the words in the second and third column of the table's title row?
    A few right brackets do not have spaces after them. 

    I know you're likely too short words for this, but I think it would be helpful to present some of the main arguments against digital products made by Hendricks, 2010; Kirkup, 2010; and Savage, 2006). This allows the reader to join in the critical thinking process in which your paper clearly has already engaged. 


    This is a complex methodology to describe with modern word limits, but is it possible to add one sentence on Framework Analysis? You explain that FA involved classifying each ed. product into Boyer's Framework--so from my reading FA is meant to allocate things into categories? My experience with FA is a tabular data analysis technique for filtering coded data, which is similar in some ways than how you've used it but different in others. I see the Gale et al. 2013 citation would undoubtedly explain Framework, but for those unwilling to probe the works cited list it might help to include "Framework Analysis is an approach that uses X for Y." 

    So by the time we reach the Combining the Framework and Thematic Analyses heading I'm wondering a few things:
    Are all 19 themes included in Table 3? 


    The scoping review process is clearly articulated. The results and discussion of this paper are excellent.

    One of the keywords, though, is tenure and promotion and this issue does not feature in the discussion at all. My sense is that this is included in order to make a case that faculty ought to be assessed on their digital scholarly product development alongside traditional scholarly product development? 


    This framework is important. I will use this framework. 

    Overall Evaluation
    If this were KeyLIME I'd give this paper a thumbs up. However, I've been asked to provide a more thorough overall assessment using a 5-star rating system. This is a 5-star paper in my opinion. The authors clearly elaborate their goals and succeed in presenting their results and how the results came to be. 

    This review has 1 comments. Click to view.
    • Placeholder
      Brent Thoma


      Thanks for your excellent review! I’ve addressed your points in order below (apologies that it is all lumped together, there doesn’t seem to be anything that I can do about that).

      Excellent points regarding the terminology. After reviewing the manuscript I see what you mean regarding it being somewhat unclear. Upon further review, I have removed all mentions of ‘digital scholarly products’ (now just called digital products) and ‘digital scholarship’ (now just referred to as scholarship). I think this is important because the thrust of our paper is that digital products are scholarship and we should not, as you note, create a dichotomy between digital and non-digital scholarship. The question is not ‘what is digital scholarship?’ but ‘are digital products scholarship.’ This has been reflected throughout the manuscript. Additionally, ‘digital products’ is now defined at the beginning of the manuscript.

      Structure & Scope: Thank you.

      Style: I believe these have been corrected. Thank you for catching these detail.

      Background: I have added a brief note of the arguments against digital products to this section and added additional information on this to the discussion.

      Methodology: Based on Isabelle and your reviews the methods section has been revised extensively. I believe the purpose and explanation of the two analyses now reads more clearly.

      I have made the text more clearly indicate that 18/19 of the products defined in the Thematic analysis were included in Table 3 and that virtual reality was the only one that did not have a parallel.

      Analysis: In our minds, the last paragraph of the Discussion focus directly on tenure and promotion with their analysis of the value of digital products in academia. I have added text making that connection extremely clear.

      Conclusion: Thanks again for this extremely helpful review! We will be sure to pass along your recommendation to the KeyLIME gurus in the hopes that they also take a look at our paper.

  • Placeholder
    Isabelle Colmers
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    In this manuscript, Thoma et al use a
    systematic search of four online literature databases as well as google scholar
    to identify 524 published works that feature the use of digital technologies in medical education.
    The authors include a graphical description of the ‘explosion’ of digital product
    publications in recent years. They applied Boyer’s Framework of Scholarship to
    classify newer published digital products, with 85% falling under “teaching
    scholarship”. A thematic analysis revealed 19 different taxonomies into which
    the authors classified digital product publications, with the most common
    category of web-based or computer-based assisted learning (41%). The authors
    conclude this new taxonomy can be applied to publications of digital products.


    Overall, the paper is well written and the
    conclusions reached by the authors are reasonable. This work is a novel
    contribution to the literature and is relevant to a larger audience of medical
    (and other) educators and scholars.

    I have no conflicts of interest to declare.


    Key suggestions:

    - My main concern is the inherent
    difficulty in capturing the breadth of digital innovations or products. Though
    the authors do address this point in their limitations, would it be possible to
    expand on this and estimate a possible impact on the study’s findings? Would
    the grey literature and unpublished works  change the taxonomy (and how)?

    - The paper heavily references Boyer’s
    Framework of Scholarship. Readers would benefit from a more in-depth
    description of this framework as well as the context in which it was published
    (so to understand why it now must be revisited). As well, a brief description
    of how the thematic analysis and constant comparator technique was used would
    be helpful to readers less familiar with these methodologies.

    - The authors conduct a systematic review
    of 4 major databases; later in the discussion a fifth database (MedEdPORTAL) is
    mentioned as a novel peer-reviewed platform for publication of digital
    products. It would be interesting for the authors to run their search using
    this platform or alternatively explain why searching this database was not

    - If possible, it would be interesting to
    see a curve fit to the explosion of literature shown in figure 2. A numerical
    description of the rate of growth would be a great addition to the paper.

    - It is noteworthy that most
    digital products were categorized under “teaching”. It would be interesting to
    have more discussion and exploration around this finding (and the sparse
    population of the other categories with the framework). Could this reflect the literature captured by the search strategy? Is this a gap in the digital publication literature
    (publication bias)? Or is there a gap in the digital medical products that exist (or have
    existed) in general?

    - It is not convincing that a systematic
    review with two reviewers and a third auditor would be meaningfully limited by
    exclusion or misclassification. The authors can consider omitting this

    Minor suggestions:

    - Commonly ‘figure 1’ of study
    flow in systematic reviews includes the number of full text studies screened.
    In this case, though not all included studies had a full text review, the
    authors can consider adding this out of convention or interest. The need for
    full text review may also indicate the degree of emphasis of the digital
    product within the publication.

    - The reference
    for Boyer may have been truncated. Google will direct to the work when using: Boyer, E. L. (1990) Scholarship Reconsidered. The Carnegie Foundation for the Advancement of Teaching, San Francisco, CA: Jossey-Bass Publishers.

    This review has 3 comments. Click to view.
    • Img 20150909 170817 edit edit
      Allan McDougall

      This is a good point and I'd also tack this on to my own review below.

      "- It is noteworthy that most digital products were categorized under “teaching”. It would be interesting to have more discussion and exploration around this finding (and the sparse population of the other categories with the framework). Could this reflect the literature captured by the search strategy? Is this a gap in the digital publication literature (publication bias)? Or is there a gap in the digital medical products that exist (or have existed) in general?"

      • Placeholder
        Brent Thoma

        Thanks Allan,
        We agree - this was a really insightful point. We addressed it to the best of our ability in the final revision, although it is hard to really know for certain what the answer is.

    • Placeholder
      Brent Thoma


      Thank you for your detailed review! This response goes through your comments point by point (apologies if the formatting lumps together, there doesn’t seem to be anything that I can do to fix that).

      Key suggestions:

      We have added a sentence to the limitations noting that the inclusion of the grey literature and unpublished works may have led us to miss digital products in our taxonomy, but that it is unlikely that these products would be of substantial importance if they had not been described in the literature.

      Further details have been added to the introduction to explain the context into which Boyer’s Framework of Scholarship came into being.

      The methods section was rearranged and some additions were made to ensure that the description of the constant comparator technique was clear.

      An explanation for the exclusion of MedEdPORTAL as a database was added to the limitations section. To explain succinctly: MedEdPORTAL publishes digital products themselves while our search focused on digital products described in the literature. For this reason, it was excluded.

      A polynomial line was found to best fit for the data in Figure 2 and an equation was calculated to represent it numerically. However, we did not find that the line or the equation added substantively to the interpretation of our data and decided not to include it.

      Additional discussion about the dominance of digital products focused on teaching and learning has been added to the discussion.

      The limitation regarding the reviewers has been removed.

      Minor suggestions:

      Unfortunately, we did not track which manuscripts required full text review and which did not so this information cannot be added to Figure 1.

      The Boyer reference has been corrected.

      Thanks again for your excellent review!


  • Placeholder
    Nishan Sharma
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    I believe the data supports the authors' conclusions.
    I have no conflicts of interest to declare.

    The authors have done well to present a
    review and analysis of a broad topic plagued with overlapping scope and


    As the paper focuses on examining digital
    products within Boyer’s framework, I believe a slightly deeper presentation of
    Boyer’s Report in the introduction would help readers understand the authors’
    approach. A brief sentence or two on Boyer’s motivation for the work, its
    conclusions and its contributions (e.g. tackling the divide between research
    and teaching) would ground its use as the foundation of this article.


    In Table 3, comparisons between traditional
    and digital scholarship seem appropriate for all but the row “Independent study
    resources”.  The single term “Coursework”
    may be expanded so as to not artificially diminish its comparison to the
    digital list.  The juxtaposition of the
    one word with the digital tools does not read as a comparison.  Independent study resources in my own
    experience means things like open lab times, access to anatomical models and
    histological slides, alternative textbooks, supplementary exercises, and access
    to tutors.


    In the 4th paragraph of the
    discussion, the authors address the advantages of digital scholarship, as part
    of their thesis to promote “the role and value of digital products in our
    current academic schema”.  A few more
    examples of the advantage of digital products (reach, alternative method of
    increasing engagement, multimodal design, control over pacing, ability to
    update) would strengthen this point.


    I question whether the first limitation is
    necessary.  Two coder analysis with a
    method for addressing disagreement is a well established methodology.  I would suggest if the authors wish to
    “defend” their approach, referencing the method would suffice. 


    I am struggling with the last limitation. I
    am unclear on what the authors mean by “terms that are not applicable today”,
    and the statement about the academic merit of such resources.  Perhaps it is citing the example of CD-ROM?


    I am uncomfortable with the phrase “advent
    of the internet”, which is generally attributed to the late 50s or early
    60s.  Another descriptor or anchoring on
    your reference to 1995 might be a better option.


    Formatting: consistently remove periods
    from examples in table 2, prior to citation. Labels in Figure 2 could be

    This review has 1 comments. Click to view.
    • Placeholder
      Brent Thoma

      Dr. Sharma,

      Thank you for providing such a helpful review!

      Both your review and Mahan’s suggested clarifying the importance of Boyer’s framework further in the introduction. We have added further context to the opening paragraph.

      Regarding Table 3, we agree with your suggestion for expanding the term ‘Coursework’ and have done so.

      Thank you for your additional suggestions regarding the advantages of digital products. These points have been added to the text.

      We also agree regarding the first limitation. It was an unnecessary thing to mention as our methodology was is consistent with best practices for this type of analysis. We have removed it.

      We agree that the final limitation was unclear and have added additional context to clarify what we mean.

      We have modified the phrasing “advent of the internet” to refer to its democratization and widespread accessibility.

      The formatting of the citations in Table 2 have been corrected. While we haven’t made the labels in Figure 2 bigger yet, this will be done prior to final publication of the manuscript.

      Once again, thank you so much for taking the time to review our manuscript! It has been improved substantially as a result of your comments.

      On behalf of the authorship team,

      Brent Thoma

      • Img 20150909 170817 edit edit
        Allan McDougall

        Ah! As I was reading these comments I was wondering why I had the feeling your summary of Boyer's Framework was actually quite detailed. Now I realize my review falls in the 2nd revision cycle here.

  • Placeholder
    Mahan Kulas
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    I'm intrigued by the following -
     - the authors mention the idea of the community of practice. The concept itself is a mechanistic in describing how practice occurs and changes to practice can take place. Do we have any evidence to suggest that CoP exists in a 'true' globally digital fashion or are they distributed local but manifested digitally?
    - More information and description of Boyer's model would be of interest to lay readers (and us psychologists who distrust all frameworks!). What does the model add? Has it been applied to med-ed previously? You provide some more detail in the methods but foreshadowing this in the intro would be valuable.

    Generally, the introduction highlights the purpose and questions of the paper though the authors could foreshadow what the value added of quantification and organization of digital products. Will it be part of a larger set of studies generating standards for these types of scholarship?

    The relationship between the Framework analysis and the Thematic analysis should be explicitly stated. Without the benefit of the table in the Results the difference between the Thematic analysis and the Framework analysis as well the purpose of doing both is not clear. Was the process of Thematic analysis similar to Framework analysis? And examining Table 2, it seems really what is being catalogued is the medium or format of the product. Why is this a theme (which has specific interpretations in other literatures).  The authors also state they will compare digital to traditional methods. What method or approach was used? Table 3 provides a second type of classification system - where was this system generated? How were the definitions created and applied?

    The methods are the weakest part of the paper. While the scoping review methods (and admittedly, I'm not an expert in this type of review) are well described more details on how the tables were generated, the analysis procedures etc. would be useful. If the comments don't apply, then provide a rationale.


    The limitations are well described. The authors note : "we noted several emerging trends. Web-based and computer
    assisted learning programs have become a prominently featured resource in
    medical education. " Was there a formal analysis of trends? The authors present a chart which tracks the rate of publication. It would interesting to see if there is a longitudinal pattern of change in the number and types of digital products published. If they had sufficient sample size, such a model would be interesting. Paradis et al. has done work on bibliometric analyses of publications in med ed (e.g. IPE) that can be helpful. Again, it depends on the sample size available.

    It would be important to highlight some of the disadvantages of digital products. For example, the evidence that they change measured learning outcomes is weak, affected by publication bias, and equivocal. This would be provide more balance to the paper.

    One of the interesting things in the conclusion is that the authors note "parallels" between digital and traditional products. So are digital tools a cheaper, more durable method of doing what we do traditionally? Or are there some opportunities that are currently being missed? In other words, are we harnessing the potential of this media fully?

    Figures, referencing and other components are appropriate. Writing and style is clear and concise.

    This review has 2 comments. Click to view.
    • Placeholder
      Brent Thoma


      Thank you for your helpful review! We have responded to each of your points in turn and, where possible, upgraded the manuscript to reflect your suggestions and noted when applicable.

      Regarding the development of online communities of practice, the literature on this is scant. The references that were cited in paragraph 2 of the introduction describe online communities tied together by academic interest that communicate primarily via their blogs. I suspect the extent to which these online communities are global versus an online manifestation of a local community will vary with the group and topic, however, I have not found research studying this. That would certainly be a question of interest.

      I agree that we could have elaborated further on Boyer’s framework. It is nearly ubiquitous in medical education; however, we should not take for granted that readers are familiar with it. Further context and a more in-depth explanation of the significance of Boyer’s work have been provided in paragraph 1.

      A more explicit statement of our goals has been added to paragraph 3. We believe your question regarding future directions is more appropriate for the discussion/future directions section and has been addressed there to the extent possible.

      The relationship between the Framework and Thematic analyses was more explicitly defined in the Methods section. The methods section was reformatted to make this clear.

      The process between the two analyses was concurrent but distinct. The most obvious difference was that the Framework analysis assessed the fit of digital products within an existing framework (Boyer’s) while the Thematic analysis attempted to derive definitions to form a taxonomy of digital products. While the Thematic analysis may not have been completely traditional, we feel that the described methodology (categorization of the products followed by qualitative assessment to develop a consensus definition that was refined using a constant comparator technique) warranted use of the term. Is there a term you think would be more appropriate for this methodology?

      We agree that how the comparison between digital and traditional methods was not made clear in our original initial manuscript. The admittedly subjective methodology used to for this purpose has been described in greater detail in paragraph 3. The information presented in Table 3 was a result of this process.

      Thank you for your assistance in clarifying our methodology. We believe our draft reads much more clearly as a result of your questions and requests for clarification.

      Trends were not formally assessed and, to avoid any confusion, we have removed this term. However, I think our statements are backed up by our data. For example, there were far more (>4 times more) web based and computer assisted learning programs described in the literature than any other digital product and social networks were the 3rd most common product but first appeared in the literature only recently.

      While we agree that a more detailed analysis of the longitudinal patterns of publication may be interesting, that is beyond the scope of what we hoped to accomplish in this manuscript. The data concerning the rate of publication was used to demonstrate the increasing number of digital products being described in the literature and not to quantify the trends in the growth and prevalence of particular products. To do this well I suspect we would have to find a way to quantify the many digital products that have never been described in the literature.

      Additional limitations of digital products have been added to provide more balance to the discussion.

      We have also added further commentary regarding the parallels between digital and traditional products and how they can be utilized, however, further study is needed to determine the optimal integration of traditional and digital tools.

      On behalf of the authorship team, thank you again for your review! It has been very helpful to us as we work to improve our manuscript.

      Brent Thoma

    • Img 20150909 170817 edit edit
      Allan McDougall

      I intentionally avoided reading the other reviews before writing my own. I think these are two excellent points by Mahan:

      "It would be important to highlight some of the disadvantages of digital products. For example, the evidence that they change measured learning outcomes is weak, affected by publication bias, and equivocal. This would be provide more balance to the paper.

      One of the interesting things in the conclusion is that the authors note "parallels" between digital and traditional products. So are digital tools a cheaper, more durable method of doing what we do traditionally? Or are there some opportunities that are currently being missed? In other words, are we harnessing the potential of this media fully?"

      • Placeholder
        Brent Thoma

        Thanks Allan!
        Those were excellent points and we've done our best to address them both in the final version. In particular, they have been fleshed out more in the discussion.
        This peer-reviewer to peer-reviewer commentary is awesome!

  • Placeholder
    Jijo Thomas
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    Digitization in education is one of the approach and scholarship can be the best initiative to encourage students for study. We have recruited some of the most talented and focused scholars and professional assignment help writers having degrees in statistics to work for us. They know how to frame the paper by including every technical detail and helpful knowledge, as and when required.

  • Placeholder
    siena william
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper
    0 is a one-stop solution for all urgent assignment help needs. We have the
    finest assignment writers from Australia for all your academic needs. Our
    assignment experts possess great skill in writing assignments. Lecturers,
    professors and industry practitioners provide impeccable writing services to
    students from K-12 to Ph.D. level.

  • Placeholder
    reiner mart
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    A changed landscape is emerging, which offers academics new ways of
    working in research and new kinds of academic output for them to use in
    their teaching. Two areas are considered: the conduct of research (in
    particular the range of activities associated with publication and
    dissemination of research findings) and the conduct of teaching. The
    links between these practices are explored in terms of a revised view of
    scholarship, the perspective of digital scholarship. The paper begins
    with a review of literature and draws on an interview study to
    illustrate changes in academic practice, both for teaching and research.
    The study of 22 academics with expertise in educational technology
    provides a commentary on the changes, identifying open access as
    important in both teaching and publication. The paper concludes with
    some suggestions for future work in this area. THere are thousads of app which offer free scholarships for students. You can get all those apps for  free on Tutuapp apk.

  • Placeholder
    Michael Cenkner
    Originality of work
    Quality of writing
    Quality of figures
    Confidence in paper

    I commend the intention of the authors to bring validation
    to the act of teaching.

    The premise of the paper however is flawed, that counting
    and categorizing things that can be used for teaching will bring that
    validation about. There’s a conflation of teaching resources with “products of
    scholarship of teaching,” while the latter is defined in the paper as “the
    systematic study of teaching and learning in the presence of learners,” i.e.,
    not learning resources.

    One might frame the whole argument differently. The
    difficulty is not that “digital products” per se aren’t valued as scholarly
    output. The difficulty is that teaching itself is not evaluated in a rigorous
    way, and therefore is not valued in the same way that other scholarly outputs

    Boyer in 1996 revisited his model in the talk, “From
    Scholarship Reconsidered to Scholarship Assessed.” In that talk, he described the
    following grid for assessing the four kinds of scholarship:

    The left-hand column contains his proposed criteria for
    evaluating each kind of scholarship. We note that the first four can be thought
    of as being instructional design.  From
    the point of view of design methodology, we could make valid inferences to
    evaluate each criterion. For example, given a goal concerning clinical
    reasoning, a (teaching) procedure of (simply) reading cases demonstrating
    clinical reasoning, would be prima facie insufficient; a practice component is
    necessary. The effectiveness of the instructional design could be inferred by peers from samples (two sources of evidence Boyer suggests). 

    Boyer is pointing to a much-needed framework for evaluation of teaching, and it is that framework that would bring some degree of objectivity and presumably validation of the scholarship of teaching. Presumably if teaching does become primary
    again, this is the kind of scrutiny that would be applied.

    This review has 1 comments. Click to view.


This article and its reviews are distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and redistribution in any medium, provided that the original author and source are credited.