Reviewer Support (Draft)

1.1. The online learning environment is inclusive

  • What to do? Skim the entire learning package quickly to see if language is used culturally appropriate and respectful of the diversity of Australia’s people. You may want to focus on the following areas while checking if inclusive language is used appropriately when referring to:

+ Aboriginal and Torres Strait Islander peoples

+ Age diversity

+ Cultural and linguistic diversity

+ Gender and sexual diversity

+ People with disability

  • Where to focus? The following sections of the learning packages that are more likely to provide evidence of appropriate and inclusive language: Announcements, Discussion Forum, Introduction.
  • What to look out for, specifically? Examples of appropriate and inappropriate language use for each of the above areas can be found in the Style Manual by Australian Government, available here: https://www.stylemanual.gov.au/accessible-and-inclusive-content/inclusive-language
  • Further information: The principles of inclusive language and further examples of good practices and what to avoid can also be found here, in the UQ Guide to using inclusive language: https://staff.uq.edu.au/files/242/using-inclusive-language-guide.pdf
  • Tip: You may recognise that examples of inappropriate and non-inclusive language often stand out and hence can be identified quite easily without careful reading of the site. While skimming the learning package, if anything catches your eyes, stop, and leave a note in the Comment section at the end of Standard 1. You may have difficulty finding it again later as supporting evidence of your evaluation in the Combined review.

  • What to do? Check if diverse perspectives are represented and discussed in the learning resources, teaching activities and assessment tasks of the learning package. Diversity can be in the form of:

+ Authors: Materials are written by different people, not an individual or single group, of a particular cultural background, gender, or age range only.

+ Theories: Various theories and viewpoints are presented and discussed e.g., in the slides and readings

+ Chronological order: Resources may cover different developmental phases of a phenomenon or artefact. If the learning package focuses on a particular period of time, e.g., contemporary TEL practices, readings may be published across a number of years within that period.

+ Geographical settings: Resources should focus on more than one geographical area e.g., not Australia only.

+ Genre: Resources provided in the learning package should be a combination of different types, e.g., journal articles, book chapters, reports by government and organisations, etc.

  • Where to focus? The best places to look for such evidence of diverse perspectives are in the Readings/Resources section, also in the class slides, course schedule, learning objectives, and also assessment task description.
  • What to look out for, specifically? See if you can find readings by different authors and of different genres, including literature review papers; content on various theories and developmental phases in the weekly schedule; a course reading list with different years of publications, and titles/abstracts that mention different geographical regions.
  • Image: The following image is taken from iStock/VictoriaBar
  • Tip: Make sure you leave a comment in the comment box explaining why you give that success indicator a Yes/Yes But/No But/ No. It will be immensely helpful when you discuss the different evaluation between reviewers in the Combined Review, especially when you cannot remember what made you tick that box a while ago.

https://engineering.stanford.edu/magazine/article/build-team-around-diverse-perspectives-not-just-familiar-skills

1.2. The online learning environment functions across devices and platforms.

  • What to do? Check if the online learning package is displayed properly on a range of electronic devices from a desktop computer, a laptop, a tablet, and a smartphone with Internet connection. The more devices, the better as a learning package may look fine on a tablet but not on a phone screen or vice versa.
  • Where to focus? No specific areas. Use a device to log in to the learning package, flick through the menu and try all tabs without paying attention to the content. Once having checked the whole site, log out and try signing in again on another device.
  • What to look out for, specifically? Images, videos, and texts, and how they are displayed on the screen of each device. Pay close attention to where there are overlapped texts and images/videos, or where audio-visual materials are not displayed appropriately. Headings, subheadings, and bullet points or font sizes and zooming are also things to consider as they may be arranged incorrectly in small screens.
  • Note: This is an admin reviewer’s task so the first and second reviewers do not have to engage with it as you will see the evaluation result and comment by the admin review when you log in to the system.
  • Tip: Try using devices that are not the latest in the range, since many students may use basic or old devices, like an old laptop with low specifications. An admin review used to use an iPad 5, an iPhone 5s, an iPhone 6s and an iPhone 11 Pro Max to log in to the learning packages, i.e., not the latest devices. Where possible, try using devices of different makes and brands.
  • Video https://pixabay.com/photos/technology-device-gadget-mockup-5519649/
image credit: https://pixabay.com/photos/technology-device-gadget-mockup-5519649/

  • What to do? Check if the learning package can be accessible on different platforms e.g., Microsoft, Android, Macintosh etc., and operating systems, e.g., Windows, iOS, Linux, etc.
  • Where to focus? Not on any particular sections. Just check the menus and pages to see if the learning package works well and is displayed properly on a platform or operating system. There is no need to read the information given in each section.
  • What to look out for, specifically? Check the menu, pages, sections, quizzes, etc. to see if they are functional and accessible.
  • Tip: The same learning package may look different in different platforms. It may not be necessary to check the learning package compatibility with all platforms as a reviewer may not have a device running Linux operating system, so it is fine to try only the two most popular ones (Windows and iOS).

  • What to do? Access the online learning package on different browsers to see if there are any problems loading or viewing it. You may try to use as many browsers as possible, including Chrome, Firefox, Safari, and Internet Explorer, etc.
  • Where to focus? More attention can be paid to videos, images, and content areas with large amounts of text, however there is no need to spend too much time on each of them. Simply click through the whole site and see if any issues arise.
  • What to look out for, specifically? When you log in to the learning package using a browser, see if it takes much time to load the website. While you click on a menu, tab, or link, pay attention to whether the site responds quickly, and things move smoothly. Also, it may be useful to see if videos and images are loaded quickly and video or audio files play without delay or lags.
  • Tip: Try accessing the online package when you are opening multiple tabs on each browser to see if the loading speed is sluggish at all. Also, you may use different browsers at the same time, at time with peak Internet traffic such as in the evening as students may have unstable connection.
  • Image https://pixabay.com/illustrations/browsers-internet-web-design-1273344/

1.3. Online learning environment meets appropriate accessibility standards.

  • What to do? Check if the course materials meet the seven principles for universal design listed below:
  1. Equitable Use
  2. Flexibility in Use
  3. Simple and Intuitive Use
  4. Perceptible Information
  5. Tolerance for Error
  6. Low Physical Effort
  7. Size and Space for Approach and Use
  • Where to focus? Check the website, the content areas and activities presented in the learning package. More attention can be paid to the learning materials, weekly activities and assessment tasks as the students will focus on these areas hence accessibility is important there.
  • What to look out for, specifically? It may be easier to focus on things that may arise as an accessibility issue for a certain group of users, for example those with disabilities. If there are any concerns or potential issues with accessibility, make sure you leave a comment to explain your opinion on the success indicator. Typical examples of accessibility issues can be found here: https://www.washington.edu/accesscomputing/AU/v2/problems.html
  • Further information: More information regarding the seven principles for universal design is available here: https://universaldesign.ie/what-is-universal-design/the-7-principles/
  • Tip: Have a quick look at the seven principles for universal design listed in the link above, then quickly check the content of the learning package. Looking for potential issues and evidence for why they can be an accessibility issue. If there is none, then a Yes can be selected.
  • Image https://www.dreamstime.com/universal-symbols-accessibility-universal-symbols-accessibility-concept-isolated-image132225710

  • What to do? Check if external tools and applications adhere to accessibility standards using seven principles for universal design provided in Section 1.3.1. Please note that you may not be provided access to all these external tools and applications by the course developer. If that is the case, please focus only on the tools and applications you can access and leave a comment about those that you cannot check.
  • Where to focus? External tools and applications may be provided in class slides, announcements, assessment or learning resources.
  • What to look out for, specifically? Like 1.3.1, it may be more timesaving to look for accessibility issues as they may be easier to identify rather than check all the course site methodically. Typical examples of accessibility issues can be found here: https://www.washington.edu/accesscomputing/AU/v2/problems.html
  • Further information: Information regarding the seven principles for universal design is available here for you to review if necessary: https://universaldesign.ie/what-is-universal-design/the-7-principles/
  • Image https://pixabay.com/illustrations/web-domain-service-website-3967926/

  • What to do? Check if the course site is optimised for screen readers. The following link contains examples of problems that may make it hard for screen readers to work efficiently: https://www.washington.edu/accesscomputing/AU/v2/problems.html Also, it is important to see if files are consistently named and organised by type and size.
  • Where to focus? Have a quick look around the whole site for checking compatibility with screen readers. For file names and sizes, you may want to check the Learning Resources section, Assessment and Announcements as they would usually be the places where files are uploaded.
  • What to look out for, specifically? Pay attention to headings, subheadings, texts, images with text, and also the naming protocol of files uploaded. Look for potential issues that may interfere with screen readers, or file names that look out of place, or files that are too large and take too long to download.

Image https://www.flickr.com/photos/accessmattersnz/45982808441

  • What to do? Check if multimedia content is provided in different formats to ensure that options are available to meet users’ needs and preferences. For example, whether photos have an accompanied text description, and audio or video recordings have subtitles and transcripts in case students did not or could not listen or watch.
  • Where to focus? Multimedia content is usually provided in Announcements, Learning Resources, and other content areas of the course site like Introduction or at the beginning of a page/section.
  • What to look out for, specifically? Images, graphic content, audio and video, and other ways to present the content of these types of multimedia in case they are inaccessible, or not displayed properly. They can be in various alternate formats, for example a PowerPoint slideshow for a video, or the transcripts of an audio file can be automatically generated or composed and embedded in the video or provided as an attachment.
  • Tip: If videos are not produced by the course team but are taken from external sources, e.g., YouTube, they may not have an accompanying transcript, but students may choose to turn on subtitles, which is a function available on the YouTube video player window.
  • Video: Here is a video from YouTube support on how to set up automatic subtitles on YouTube videos: https://www.youtube.com/watch?v=Y7W41VMxyQE

1.4. Learners have opportunities to provide feedback.

  • What to do? Check if students can provide immediate feedback after they engage with learning materials provided in the course site. For example, if there is a checklist for them to mark a task as completed, a thumbs up/down button to show their opinion on a reading, a flagging option for them to mark content of importance or interest. It may also be instructions for students to leave reflection on the Discussion Forum, or comment using videos in VoiceThread, etc.
  • Where to focus? Students are usually asked to provide feedback by course staff in the Discussion Forum, Announcements, Introduction or Learning Resources sections of the learning package.
  • What to look out for, specifically? Any opportunities for students to provide their feedback, in any forms, e.g., a survey, or reflection journal, or a sentence in an announcement saying students should write to the course convener should they have any questions or feedback regarding the course content.
  • Tip: Students may have the opportunity to provide feedback directly to the teaching team in face-to-face sessions or video conferencing like Zoom meetings. However, these communications are out of the scope of the review so reviewers may only provide an opinion based on what is available on the course site. A comment may be useful to remind the course team that they can provide explicit instructions in the course site on ways students can provide immediate feedback to teaching staff. This would be helpful in ensuring that students know what to do to provide feedback in case oral instructions were missed.
  • Image https://pixabay.com/illustrations/feedback-report-back-balloons-4746811/

  • What to do? Check if there are any instructions or encouragement for students so that they can provide their feedback to teaching staff at different points of time, i.e., in different weeks at the beginning, middle and end of the course. The feedback can be of any form, e.g., orally or in writing, and various formats, e.g., via email, in Discussion Forum, feedback survey, etc.
  • Where to focus? Such instructions for students to provide feedback may be found in the Announcements, Introduction, Discussion Forum or even in Assessment section.
  • What to look out for, specifically? This may sometimes be a short sentence somewhere in the course site introducing the learning package to students or inviting feedback from them in sections mentioned above. You may also want to look for links to background surveys, reminders to complete course evaluation, discussion threads, reflection journals or even a quick feedback page in the weekly lecture/tutorial slides.
  • Tip: Pay attention to critical points of time like the first and last weeks, and probably around the mid-semester break. If you can find instructions requiring students to provide feedback at about 2-3 different points of time or more during the whole course, then it probably deserves a good Yes.
  • Image https://pixabay.com/illustrations/devops-business-process-improvement-3148393/

  • What to do? Check if there are the instructions provided to students on how their feedback should be submitted and what it would be used for.
  • Where to focus? Like in 1.4.2, this may be found in the Announcements, Introduction, Discussion Forum or even in Assessment section
  • What to look out for, specifically? The instructions on feedback collection and usage can be given in texts, or orally. You may want to look for the description of purpose in a feedback or evaluation survey, or the introduction of a completion checklist for learning materials, a discussion thread opening, or task description of a reflection journal, etc.
  • Tip: Such instructions may be explicit or implicit, for example students may be asked to click on a star rating, place a heart or leave a thumbs up in Padlet without clear explanation why it should be done. Alternatively, students may see a thumbs up / thumbs down button next to a reading or smiley emoticons after an activity without instructions but assume that they are expected to click on one of them to show their opinion and attitudes. Also, students may be informed of how they should provide feedback but not of how it would be used. In such cases, a Yes But or No But would be a more feasible outcome than a Yes.
  • Image https://pixabay.com/photos/feedback-survey-nps-satisfaction-3709752/

Comments

2.1. The navigation and layout of the online learning environment is functional, consistent and intuitive.

    • What to do? Skim the learning package site to see if there are instructions on the key content areas and where to find them.
    • Where to focus? These instructions may be provided in the welcome announcement, introduction section of the online learning package, the course profile or alternatively in the support guidelines.
    • What to look out for, specifically? Such instructions can be written in text e.g., a few sentences telling students where to look for things, or in audio-visual forms like part of the introductory video.
    • Tip: If there are instructions on either course site navigation or learning activities but not both, you may want to consider giving the success indicator a Yes But.

  • What to do? Skim quickly through the course site to see if it is presented in a consistent style across the sections. What matters is not what style the learning package follows but the consistency throughout.
  • Where to focus? You may want to check the Introduction section where the course structure is presented, the menus/tabs of the site, and flick through the main sections to see if they are formatted and organised consistently.
  • What to look out for, specifically? Attention may be paid to inconsistency as they may be more easily identified, e.g., in language conventions, referencing, or presentation.
  • Further information: The Style Manual by the Australian Government is available here for your information on what to look out for: https://www.stylemanual.gov.au/
  • Tip: You do not need to read the content of the learning package but pay more attention to heading levels, overall formatting, and structure. You may also want to focus on the content areas where there are regular posts or materials that are uploaded at different times and they are most likely places where there are inconsistencies in style.

  • What to do? Click all links provided to see if they work. If any of the links are dead, take note of them so the course team know exactly what to fix and where to find them.
  • Where to focus? Links are usually provided in learning materials, course reading library / reading list, in the support section and also in the introduction or announcements or course updates.
  • What to look out for, specifically? Links are easily identifiable as they are embedded in texts, however care should be taken so that all links are tested.
  • Tip: You may want to take note of the location of the link and copy/paste the dead links into the Comment section to facilitate the course submitter’s double checking and replacing them with functional ones.

  • What to do? Check if there are instructions on whether learning resources are accessible right on the course site or elsewhere, e.g. on the library, the publisher’s website or a bookstore.
  • Where to focus? Such instructions may be found at the beginning of the section for learning materials, introduction to the course, announcements, or course profile.
  • What to look out for, specifically? Explicit information on where learning resources are available and whether they are on the online learning environment.
  • Tip: The information may be brief and appear only once or a few times in the course site to avoid repetition, therefore it may be unnecessary to check if such instructions are provided for each reading or material type.

  • What to do? Check all the external resources like embedded links to see if they open in a new window or different tab from the one learners are viewing.
  • Where to focus? Most of the external resources may be provided in the Learning materials/Resources/Readings section, but they may also be in the weekly content folder or announcements/updates.
  • Tip: You may want to take note of any inconsistencies i.e. all links are open in the same window, and provide these links in the Comment section for Standard 2.

  • What to do? Check if the teaching team has described all the channels that will be used for communicating with their students, for example how long they should wait for a lecturer’s email response or when assessment results will become available.
  • Where to focus? You may want to check the Announcement, Introduction, Discussion forum and Assessment sections. The information may also be provided in the slides of the first lesson or welcome and course introduction video.
  • What to look out for, specifically? Text or audio-visual instructions on how long students should wait for a response or performance feedback, in what form the feedback or correspondences would arrive. The frequency of communication and course updates are also good examples for this success indicators.
  • Tip: If not all of the necessary information can be found, a Yes But may be more suitable than a No But, which may be given when the information provided is implicit or unclear.

2.2. The online learning environment is logically sequenced and organised

  • What to do? Skim the learning sequence to see if the content or topics are scaffolded coherently and built upon previous learning.
  • Where to focus? The same areas as in 2.2.1., with special attention paid to the course structure presented in the course profile or learning resources section.
  • What to look out for, specifically? The focus should be on the course syllabus or curriculum and how the key learning content areas are organised.
  • Tip: If the course content is not coherently structured or a learning component/topic is not built upon the content and knowledge that have been provided previously in the learning package you may want to go for a No But or a No dependent on how often this happens.

  • What to do? Check to see if the learning content is scaffolded appropriately so students are not overwhelmed or find the workload insufficient. The headings and subheadings should also be looked at to ensure they reflect the learning content accurately and succinctly.
  • Where to focus? You may look at the Learning resources, course profile and assessment sections.
  • What to look out for, specifically? Examples of chunking may be shown in consistent amounts of work assigned to students and the amount of time students are expected to spend on learning each week. You may also look at the headings and subheadings for learning segments/chunks too.
  • Tip: What is meant by “manageable” and “appropriate” may mean different things for different people, therefore you may want to consider the workload assigned in relation to the course level and students’ background.

  • What to do? Have a look at the overall content presentation and see if it is easy to navigate around the learning resources provided.
  • Where to focus? The overall site structure, menus, or pages, and most importantly the learning resources section.
  • What to look out for, specifically? The way the course site is organised and how key content areas are presented.
  • Tip: It may be easier to look for issues with navigation around the main sections and learning resources, and make sure you describe that issue in the comment section.

There are some guidelines for the comment section at the end of Standard 1.

Comments