From 2663e21be32433b1184f29f2d9e68298e2c733c1 Mon Sep 17 00:00:00 2001 From: "Eloy Lafuente (stronk7)" Date: Thu, 17 Mar 2022 23:48:39 +0100 Subject: [PATCH 1/2] MDL-72698 behat: Reduce random scorm behat failures on slow envs This just applies two "tricks" to make the test to behave better: - Ensuring that the "wait until the page is loaded" happens in the scorm frame. - Ensuring that the last item in the package gets some extra time by reloading the page completely at the end. With them, a testing environment with a 100% of failures has become 100% passing (25 repetitions). That should be good enough to reduce the impact of that failure everywhere. See the issue for more details. --- .../behat/scorm_activity_completion.feature | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mod/scorm/tests/behat/scorm_activity_completion.feature b/mod/scorm/tests/behat/scorm_activity_completion.feature index 059ae7330f2..68be9617ce4 100644 --- a/mod/scorm/tests/behat/scorm_activity_completion.feature +++ b/mod/scorm/tests/behat/scorm_activity_completion.feature @@ -52,26 +52,48 @@ Feature: View activity completion in the SCORM activity And I press "Enter" And I switch to the main frame And I click on "Par?" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "Keeping Score" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "Other Scoring Systems" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "The Rules of Golf" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "Playing Golf Quiz" "list_item" And I switch to "scorm_object" iframe + And I wait until the page is ready And I click on "[id='question_com.scorm.golfsamples.interactions.playing_1_1']" "css_element" And I press "Submit Answers" + And I wait until "Score: 20" "text" exists And I switch to the main frame And I click on "How to Have Fun Playing Golf" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "How to Make Friends Playing Golf" "list_item" + And I switch to "scorm_object" iframe And I wait until the page is ready + And I switch to the main frame And I click on "Having Fun Quiz" "list_item" And I switch to "scorm_object" iframe + And I wait until the page is ready And I click on "[id='question_com.scorm.golfsamples.interactions.fun_1_False']" "css_element" And I press "Submit Answers" + And I wait until "Score: 33" "text" exists + And I switch to the main frame + # We need to get some time till the last item is marked as done (or it won't be ready in slow databases). + # This could be a pause of a few seconds (working ok in super-slow oracle docker database), but forcing a + # reload seems to be doing the work too under that very same slow environment. + And I reload the page + And I wait until the page is ready When I am on the "Music history" "scorm activity" page Then the "View" completion condition of "Music history" is displayed as "done" # Conditions that are not possible to achieve (eg score below requirement but all attempts used) are marked as failed. From 9392313b8844fc9c52d14e853c7249b8a1496650 Mon Sep 17 00:00:00 2001 From: "Eloy Lafuente (stronk7)" Date: Sat, 19 Mar 2022 14:50:40 +0100 Subject: [PATCH 2/2] MDL-72698 behat: Reduce random lesson behat failures on slow envs In very slow environments viewing the lesson page for a student takes more than 1 second and, because of this, the "Spend at least 1 sec" completion condition is considered done. So, in order to alleviate this problem, that leads to random failures in CIs with slow databases, we have changed the condition to be 3 seconds instead. That seems to do the work in super-slow environments, with 100 executions passing ok (when previously the scenario was failing 33% of times). The only cost is that now the behat run will be 2 seconds slower (there is a wait changed from 2 to 4 seconds), but that's an acceptable price to pay to get the scenarios consistent passing. See the issue for more details. --- .../tests/behat/lesson_activity_completion.feature | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mod/lesson/tests/behat/lesson_activity_completion.feature b/mod/lesson/tests/behat/lesson_activity_completion.feature index 6747e18469e..fb53a4e9abc 100644 --- a/mod/lesson/tests/behat/lesson_activity_completion.feature +++ b/mod/lesson/tests/behat/lesson_activity_completion.feature @@ -27,7 +27,7 @@ Feature: View activity completion in the lesson activity | completionusegrade | 1 | | completionendreached | 1 | | completiontimespentenabled | 1 | - | completiontimespent | 1 | + | completiontimespent | 3 | And I am on the "Music history" "lesson activity" page logged in as teacher1 And I follow "Add a content page" And I set the following fields to these values: @@ -49,28 +49,28 @@ Feature: View activity completion in the lesson activity Scenario: View automatic completion items as a teacher When I am on the "Music history" "lesson activity" page logged in as teacher1 Then "Music history" should have the "View" completion condition - And "Music history" should have the "Spend at least 1 sec on this activity" completion condition + And "Music history" should have the "Spend at least 3 secs on this activity" completion condition And "Music history" should have the "Go through the activity to the end" completion condition And "Music history" should have the "Receive a grade" completion condition Scenario: View automatic completion items as a student Given I am on the "Music history" "lesson activity" page logged in as student1 And the "View" completion condition of "Music history" is displayed as "done" - And the "Spend at least 1 sec on this activity" completion condition of "Music history" is displayed as "todo" + And the "Spend at least 3 secs on this activity" completion condition of "Music history" is displayed as "todo" And the "Go through the activity to the end" completion condition of "Music history" is displayed as "todo" And the "Receive a grade" completion condition of "Music history" is displayed as "todo" When I am on the "Music history" "lesson activity" page - And I wait "2" seconds + And I wait "4" seconds And I reload the page And the "View" completion condition of "Music history" is displayed as "done" - And the "Spend at least 1 sec on this activity" completion condition of "Music history" is displayed as "done" + And the "Spend at least 3 secs on this activity" completion condition of "Music history" is displayed as "done" And the "Go through the activity to the end" completion condition of "Music history" is displayed as "todo" And the "Receive a grade" completion condition of "Music history" is displayed as "todo" And I press "The history of music part 1" And I set the field "Your answer" to "Some drummers play with their sticks flipped around" And I press "Submit" Then the "View" completion condition of "Music history" is displayed as "done" - And the "Spend at least 1 sec on this activity" completion condition of "Music history" is displayed as "done" + And the "Spend at least 3 secs on this activity" completion condition of "Music history" is displayed as "done" And the "Go through the activity to the end" completion condition of "Music history" is displayed as "done" And the "Receive a grade" completion condition of "Music history" is displayed as "done"