When there are many files inside a single resource, it's possible
for the backup to time out because it does not report progress
while adding entries to the backup_ids table.
This commit contains three changes in the three files:
1. A bug in the backup process meant that anything logged after a
certain point did not appear in the on-screen display of the
backup log, because the logger was serialised and deserialised
but display code referred to the old version. Changed so that
code retrieves new object.
2. Add more information to backup log when there is a missing file.
3. Add more information to restore log when there is a missing file
(and remove existing code duplication of the current message).
The 'missing file' situation is one that generally shouldn't occur in
normal usage, but when it does happen, it is useful to have full
information about the file.
When restoring a backup that contains a large number of users
to a different server where those users don't already exist,
creating users can take a significant time. This change adds
progress reporting so that it doesn't time out and shows
activity in the user interface while doing the creation.
Adds 'Preparing page display' progress bars for user interface pages
if they take a long time to display.
Also adds changes where other parts of the backup progress timed out on
long backups. After this change, and MDL-41838, it is finally possible on
my dev server to successfully back up the 'XL' test course.
1. Changes progress bar code to allow headings for progress bar (so users have
some clue what's going on if a page has more than one progress bar).
2. Changes restore code so that a progress bar can display during pre-checks if
they take longer than 5 seconds.
3. Changes pre-check and restore code so that, in various points where the system
can take a long time within an individual step, intederminate progress is
indicated and it won't time out.
Adds calls to the new backup progress tracking API within various steps of the
backup system - previously it only tracked progress between steps, but some steps
can themselves be slow. This ensures the system displays progress (either by
moving the progress bar if possible, or by making the wibbler below it pulsate)
during nearly all of the backup process.
encode_backup_temp_info() and decode_backup_temp_info() have been
introduced to keep the info field encoding in one place.
Many locations used get_backup_ids_record() to obtain info, that
makes lots of calls to get_backup_ids_record() which can be slow.
We now complete all those inline by adding the info field to the query.
To reduce memory usage, all queries of that nature have been changed
to use get_recordset_*. gzcompress was introduced if available to minimize
traffic to/from the database and to decrease the memory required for caching.
The compression time is saved by the benefits in other places of having smaller data.
The cache is function local and testing against a large database
indicates 10k questions is a large category. Restore already
uses MEMORY_EXTRA and that will have enough space for the couple
of megabtyes a local sql hash will introduce.
There were still some differences between the php code
used in the issue and the php code generated by the
xmldb editor for the template tables. This commit put
both 100% in sync.
When we import courses or duplicate activities, the deduplicating nature of
the file_storage system means that we don't actually have to include the
real files in the backup - just their metadata from the file table.
The restoration process needs to know not to expect files from the backup
phase so a flag is set in moodle_backup metadata.
Dan Poltawski