instead of
Checking partition "Microsoft Basic Data"...
Checking partition "EFI System"...
show
Searching EFI partition...
EFI partition found at /dev/sda1
and refer to EFI partition consistently in logs.
if image already exists in the cache, skip a second checksum validation.
log shows duplicated entries:
Verifying checksum for example.img, please wait...
Checksum is OK for example.img
Verifying checksum for example.img, please wait...
Checksum is OK for example.img
because tip_check_csum() is called twice in this case.
Restoring an updated image file (with different checksum) could fail while the
old checksum file could remain in place.
Remove image and checksum file before fetching the new files.
.full.sum file in the local cache contains the local checksum for this file,
this checksum is calculated by tip_write_csum(), therefore, there is no need to
calculate this checksum again from the image file in the cache, use this
checksum content instead.
if .full.sum is not available, then cache is inconsistent, tip_check_csum()
is called after checking if image file exists, raise an exception.
checksum could mismatch in two situations:
a) new image version (checksum is different)
b) image is corrupted
in both cases, a new fresh image needs to be retrieved.
this speeds up checksum validation.
Add function to redefine the hostname of a Windows install.
Windows hostnames can't be larger than 15 characters due to
legacy heritage.
Hostname modification is done by modifying 3 registry values in
the ControlSetXXX entry of the SYSTEM hive. ControlSet001 is
generally the entry to be edited but one must query the value
of the 'Current' key in the 'Select' entry of the SYSTEM hive
to retrieve the active ControlSet.
The hostname has to be introduced in the following entries:
path = 'ControlSetXX/Control/ComputerName/ComputerName'
key = 'ComputerName'
path = 'ControlSetXXX/Services/Tcpip/Parameters'
key = 'HostName'
key = 'NV Hostname'
The value to store in those keys is of tipe SZ and has to be
encoded in UCS-2 Little Endian (utf-16-le).
Replace ConfigureOs script by native calls to:
- update BCD via hivex using bcd.py and winreg.py infrastructure.
- restore efi bootloader restore_windows_efi_bootloader().
Call legacy scripts for remaining postinstall procedures to
replace them incrementally.
Define variable CONFIGUREOS_LEGACY_ENABLED as False by default.
Run legacy configureOs when CONFIGUREOS_LEGACY_ENABLED = True.
This serves as a auxiliar method to keep the restoration functional
in case of problems with the new configure_os logic.
Implement update_bcd() as replacement of ogWindowsBootParameters.
The legacy function modified the BCD of a freshly restored system
invoking the privative binary "spartlnx".
The script edits a set of BCD entries needed for a proper system
boot.
Each main BCD entry is identified by an UUID and contain a set of
subnodes, these hold the configuration and entry information.
Each node contains data in the form of key-value.
Common BCD structure:
'Objects'
...
The Boot Manager entry is always identified by the UUID
9dea862c-5cdd-4e70-acc1-f32b344d4795.
Some entries always have the same UUID as identifier such the
Boot Manager while other have different UUID depending on the system.
To identify these entries with a not known UUID we query the value
'Type' of the node 'Description'. This contains a 32 bit value whose
bytes codify the nature of the entry.
We obtain 3 different values as a tuple, each value is the result of
applying a bitmask to the Type value. These masks are 0xf0000000,
0x00f00000 and 0x000fffff. The resulting tuple of 3 values is then
used to obtain the corresponding entry from a map.
The data we modify from the BCD are disk and partition references
to point to the new disk and partition in the system where the
images are restored. Partition and disk information is stored as
UUID in specific offsets inside binary data in the BCD.
To update these we need to obtain the disk and partition UUIDs,
then convert it to bytes as follows:
original UUID: C4C61C51-3456-4733-96AD-AE760A41C392
UUID as bytes: 51 1C C6 C4 56 34 33 47 96 AD AE 76 0A 41 C3 92
The modified entries are: Resume from Hibernation, Windows Boot
Loader OS, Windows Boot Loader Recovery, Windows Recovery,
Boot Loader Settings, Windows Boot Manager and Windows Memory
Diagnostic.
Some of these options could be omited as the system restoration
does not include a recovery partition so in this case all the
recovery related entries just point to the main system partition.
Most entries are edited modifying the value corresponding to the
key 'Element' in 2 subnodes of the 'Elements' node.
These subnodes are '11000001' and '21000001'.
'Objects'
The 2 values stored in these 2 entries is binary data where we
store the partition and disk bytes. We simply replace the byte
representation of out UUIDS in the binary data. Partition is
stored in offset 32 and the disk in offset 56.
The exception is the Bootloader Recovery entry, in which the
partition offset is 84 and the disk offset 108.
Note that the legacy function only does a proper BCD edit in UEFI
systems. The new implementation follows the same behavior with
the possibility of implementing BCD modification under MBR
partitions in the future.
Set the field 16000009 (RECOVERY_ENABLED) to the value x00 to
disable recovery in both Resume from Hibernation and Windows
Boot Loader OS entries in the BCD.
The system install does not include any recovery partition so
it makes no sense to have it enabled.
This commit is preparatory work for the new native postinstall code.
Add winreg.py to the utils folder. Implement hive enum types,
hive handler validation and validated get_* functions for
nodes and registry values.
Implement the utility hive operations through the Hivex library.
This serves as preparatory work for BCD manipulation but it also
has potential to improve registry usage in previous code.
UCS-2 Little Endian is the prefered windows registry text encoding
for binary content. Define a WINDOWS_HIVE_ENCODING global variable
to use when encoding string to write in the win registry.
This commit is preparatory work for the new native postinstall code.
Add restore_windows_efi_bootloader to copy the EFI loader from
the filesystem in the restored system into the EFI partition.
This commit is preparatory work for the new native postinstall code.
Add get_partition_id and get_disk_id to obtain the UUID of a disk
or partition as a string. These values are obtained invoking the
program blkid as a subprocess.
This commit is preparatory work for the new native postinstall code.
If formatting fails, log shows:
mkfs.ntfs reports return code 1 for /dev/sda2
but ogclient reports success to the ogserver.
Raise an exception so front-end gets an indication that formatting has failed.
When checksum is not available, it displays:
(2024-07-15 09:04:14) ogClient: [ERROR] - URL error when fetching checksum: Not Found
"URL error" is leaking an internal implementation details, reword this report.
Add explicit check for .full.sum after downloading it.
Rewrite errors log, one of them is misleading when checksum validation
fails, it refers to missing .full.sum, but it could be a different
reason.
Try to find the script to run for a shell/run request in
/opt/opengnsys/shell/, restricted mode is enabled if the
script is found.
Excute the script without shell=True and executable=OG_SHELL in
restricted mode.
Restricted mode is a safer execution method as it only executes
code manually defined by the administrator.
Each script needs to define a shebang, this way more than just bash
is supported.
provide return code as result to ogserver.
Update virtual mode driver to return dummy value, although this command
is unimplemented, this seems to be broken due to possible TypeError when
accessing result from caller.
Properly report the lack of cache partition when restoring an
image using TIPTORRENT and UNICAST as methods.
Abort any restore in case of no cache partition.
Fix image size, permissions and creation time.
Improve error report related to these parameters now showing the
exact cause of the problem if any occurred during the definition
of image size, file permissions or image creation time values.
Use the constant OG_CACHE_IMAGE_PATH from cache.py to obtain the
location of the directory where images are stored.
This way the path can be changed from one single point.
Add a 'cache' field into the json payload the client sends to
the server after a restore operation so the server can update
the new cache contents.
Resquest response structure:
{
...
'cache': [
{'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'},
{'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'}
]
...
}
Add API REST method to delete cache contents.
Resquest payload structure:
{
'images': ['windows.img', 'linux.img']
}
The client will try to delete as many images in cache as available
with names matching the list of filenames in the 'images' field.
Resquest response structure:
{
'cache': [
{'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'},
{'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'}
]
}
Add the list of images in the client's cache partition in the
payload sent to the server.
The information sent is a list of {image_name, img_size, checksum}
elements where img_size is the size of the respective image in bytes.
Resquest response structure:
{
...
'cache': [
{'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'},
{'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'}
]
...
}
Rename OGIMG as OG_IMAGE_PATH.
Rename OGCACHE_MOUNTPOINT as OG_CACHE_PATH.
Define OG_CACHE_IMAGE_PATH as OG_CACHE_PATH + OG_IMAGE_PATH.
This will serve to have a unique point to obtain cache related
paths.
This method reports the /dev path to cache partition, rename it.
Add explicit check if blkid is successful.
And add logging to report that device path to cache is not found.
Add exception checks to the os.mkdir operation and log the error
found. The previous implementation was too optimistic and only
handled mount related errors.
Report mkfs failure for every partition. This does not raise an
exception as that would skip partprobe operations and the mkfs
operations in the next potentially well formated partitions.
ogCP expects LINUX-SWAP to specify a swap filesystem.
Add a similar workaround to the one that is done for VFAT for symmetry between
inputs and outputs that circulate over the API.
partprobe requires that all disk partitions are unmounted.
partprobe needs to be called to report the OS that the partition table
has changed, otherwise ogclient reports incorrect partition information.
iterate over the partition list and mount cache after partprobe is
called.
tip_client_get raises the proper error exceptions but the except
block in _restore_image_tiptorrent overwrites the reported error.
Move the raise statements in _restore_image_tiptorrent outside
of the except block.
Use "Microsoft basic data partition" to store FAT32 in case of GPT partition
scheme and 0xB according in case of MBR partition scheme according to
documentation.
otherwise error path uses uninitialized variable
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 112, in ogReduceFs
return ret
UnboundLocalError: local variable 'ret' referenced before assignment
Reduce the scope of the try except block that controls the case
of deleting the image backup in case of error. Now it only covers
the section of code after backup creation and up to image
verification. Check when the Exception is an OgError to raise
with added context.
Prevent the deletion of the target image in case of error before
the backup creation.
Bundle the backup creation on its own try except block to give
more feedback on a failed backup creation.
Enables a better error management allowing unhandled
exceptions to be reported properly.
Capture only the relevant exception types in each except block.
The capture of the Exception type means hiding information for
unhandled error cases, even for syntax errors in the codebase.
Using a more fine grained exception filtering improves error
traceability.
Log an error message in known error cases and log a backtrace
otherwise.
Define a new error type OgError to be used in all the 'raise'
blocks to define the error message to log. The exception
propagates until it reaches send_internal_server_error() where
the exception type is checked. If the type is OgError we log
the exception message. Logs the backtrace for other types.
The initial error implementation printed a backtrace everytime
an error ocurred. The next iteration changed it to only print
a backtrace in a very particular case but ended up omiting too
much information such as syntax errors or unknown error context.
The actual implementation only logs the cases we already cover in
the codebase and logs a bracktrace in the others, enabling a
better debugging experience.
Refine 97647c32aa utils: add enforce_gpt argument to get_efi_partition()
to provide more explicit error when trying to boot Windows UEFI from DOS
partition.
This reverts commit 57787dab54.
Read from stderr is blocking if no data is available, revert this patch since
ogClient hangs indefinitely in lzop invocations due to races in process
execution through Popen.
Replace unexistent mountpoint variable to report a failed
mount operation before an OS probe from a partition.
Improve the semantics of the error message replacing 'at' with
'into'.
Remove the period at the end of the log message.
Restore image file from .ant to original file name if new image
creation fails. Remove new imagen and move the .ant image file in
place of the original as previously an error meant a rename of the
image file without a revert to keep the image available.
Make init_cache() use the actual cache mountpoint returned by the
function mount_cache() for the creation of the cache directories
instead of a hardcoded path.
Implement a Python equivalent of ogCopyEfiBootLoader as the
function copy_efi_bootloader. This function copies the contents of
the folder of the EFI loader in the ESP into a ogBoot folder at
the root of the partition target of an image creation.
copy_efi_bootloader is a Windows only functionality.
Control non 0 returncode of the lzop and partclone subprocess
in image creation and restoration because this means that either
lzop or partclone has failed.
The implementation must cover cases such as not enough storage
space and log errors into /tmp/command.log and the log file of
the client handling the request.
Check the returncode of lzop and partclone subprocesses and
log the stderr of the process reporting non zero returncode.
Log the whole context of the error when an exception happens.
The previous exception handling was hidding important information
about the cause of the error.
The Windows bootloader only supports a UEFI boot from a GPT
partition. Set enforce_gpt to True in every codepath related to
Windows. When enforce_gpt is set to True get_efi_partition()
raises an exception when an MBR partition scheme is detected.
Make is_uefi_supported() only check for /sys/firmware/efi as
get_efi_partition() will detect a missing ESP or an invalid
partition scheme. Stop using get_efi_partition() inside
is_uefi_supported() as the former is eventually called in every
UEFI related code.
UEFI supports both MBR and GPT as partition schemes and this is
a required change to handle the particular case of Windows not
being able to boot UEFI from a MBR partition scheme.
OpenGnSys Support Team
Alejandro Sirgo Rica