Video assessment modules now has the ability to support auto-marking of Video assessments and provide a 'Machine analysis score' for Administrators and Learners. This allows learners to review their video assessments before submitting and admins the ability to review the videos submissions alongside the machine analysis scores to decide on final scores, feedback and completion statuses.

How to enable this feature?

An Account owner navigates to Account > Litmos Feature > Video Assessment feature to enable this feature. Admin has the option to turn the feature on or off for individual video assessment module on need basis. 

Admin experience

Administrators will be able to 'enable video analysis' on video assessment modules and and set keywords to include and avoid at the module level.  Admin can now set the max time limit up-to 15 minutes. Keywords should be entered as separate single words and should be separated by commas with no sapce.

NOTE:

• Once the AI setting for a module is enabled or disabled it is not recommended to update the settings as this will affect the learner's experience who are already assigned to that module.
• A sentence will not be considered as a single keyword.

Once learner submits the video the administrators will be able to access submitted videos along with the machine analysis scores for each submission. The Administrator can then review the videos alongside the machine analysis scores to decide on final scores, feedback and completion statuses for the submissions. Once the Admin completes the markings, the machine analysis will not be needed and is no longer available to review for those video assessment attempts

NOTE:

• Machine analysis for Video assessment is only available to the reviewers that are logged into the system. 
• Once reviewer completes the markings, the machine analysis will not be available to Admin for those video assessment attempts.

Marking Required

The completion trigger will occur when an admin marks the module through the Marking Required feature from within Litmos. 

Marking Video Assessments can be found in:

• Dashboard
• Marking column in reports
• Next to the module within the course

When no option is selected the marker can rate the video uploaded.

When Enable Review is selected, the marker can add feedback and rate the video supplied. Review allows multiple Administrators to rate and provide feedback. This is averaged for the Overall rating.

When Enable Marking is selected for the Video Assessment, the marking guide text is displayed and the marking document can be accessed. The learner is given a mark out of 100 (%) , in this example 80 out of 100. The Marker can also add feedback and mark the question as correct/incorrect.

This is the learner view of a marked video assessment with ratings (Enable Marking selected by Admin) displaying the feedback, mark and pass or fail status.

Please Note: The marker will determine a fail or pass. This is not automatically determined by the system as different assessments may require different standards or levels of competence in order to be deemed sufficient to pass.

When the assessment  is an AI Video Assessment, the AI analysis will also be displayed in each case.

Learner experience

Learner can record or upload their video assessment module and self- review the video before submitting for Admin review. 

Once learners record or upload their video assessments and click “analyze video”, learners will be prompted to ‘analyze and continue the course’ to initiate and allow machine analysis time to process the transcript.

Note: A 15 minute video will take on average 7-10 mins process but could take longer in some instance.

The learner will have to navigate back to the module listing within the course to view the machine analysis once the analysis is complete. Learner will see the machine analysis progress as 'Video analysis processing' or 'Video analysis complete' which is auto refreshed every few seconds so the learners see the latest update.

AI analyzes the learner’s transcript for emotional sentiment, rate of speech and keywords that the admin has configured.  

• Machine Analysis Score is sum of
1. Keywords to Include : Checks to see if the transcript has all the configured keywords to include.Partial keyword matches will not be included in the Machine analysis score. Scores based on unique keywords to include in transcript divided by keywords to include defined by Admin
2. Keywords to avoid: Checks to see if the transcript has any of the configured keywords to exclude. Partial keyword matches will not be included in the Machine analysis score. Scores based on total keywords to avoid minus unique keywords to avoid in transcript divided by total keywords to avoid
3. Rate of speech: Checks the rate of speech on the video and assigns the scores based on the tier values. Note: The tiers are based on general best practice and considers the conversational speech as the ideal range (120 to 180 WPM).

• Overall Sentiment  - Checks the tone of the Learner’s transcript to determine the Learner's sentiment and reflects on the progress bar.

Note:

• Translations are currently only supported for English language.
• For Mobile users the AI Video Assessment feature is only available on the on mobile browsers, it is not available SAP Litmos App .
• Partial keyword matches will not be included in the Machine analysis score