Add UEFI related utilities inside a new utility module: uefi.py
_check_efibootmgr_json
======================
Check if the system efibootmgr executable supports JSON output. This is
a private function used only by other functions from uefi.py.
is_uefi_supported
=================
Check if the system supports UEFI firmware.
run_efibootmgr_json
===================
Runs efibootmgr with json output support. Return the JSON output as a
Python dict.
efibootmgr_create_bootentry
===========================
Create nvram boot entry. This bootentry is usually later set to boot
next just once via "BootNext" nvram variable.
efibootmgr_delete_bootentry
===========================
Delete a nvram boot entry. Used to avoid duplicates when booting the
same disk and partition from a given client.
efibootmgr_bootnext
===================
Set nvram "BootNext" variable to a given boot entry so after client
reboot, PXE is not executed and the given boot entry takes precedence.
Add dependency with efibootmgr version >= 18, and efibootmgr JSON output
which is currently out of tree from util-linux repo.
Add a basic OS family enumeration: OSFamily.
Add utility function that probes for an installed Linux or Windows
system, returns the corresponding enum value, OSFamily.UNKNOWN
otherwise.
Add utility function inside disk.py to find, if any, the first ESP
partition of a given disk.
The disk is provided as an integer (starting at 1 following OpenGnsys
scripts usual values), meaning the (n-1)th disk from the disk array
returned from get_disks(). In the future a better mechanism should be
put in place to fetch probed disks from a running client.
This change is part of the upcoming drop of "IniciarSesion" script in
favor of a Python native approach. Specifically regarding UEFI systems.
value extraction did not have error checking and was handled in
a one-liner. The actual implementation expands the parsing logic
and moves it into a function.
Revisit 5056b8f0d5 ("fs: validate ntfsresize dry-run output") that has
introduced a possible infinity loop.
Disentangle this loop while at it: iterate until best smallest size is
found by probing.
do not return the returncode, instead return an integer.
do not use
except CalledProcessError as e:
it causes a another exception while handling exception.
Remount the original image repository.
it should be possible to simplify this further by:
- stacking mounts, no need to umount initial repo and mount it again
when switching to the new repo, because remount back initial repo
might fail (!)
- use check=False and simply check for x.returncode
Remove mbuffer, this is never used.
mbuffer has been never been used since ogClient supports native image restore.
Originally this was used like this:
partclone [...] | mbuffer -q -M 40M | lzop [...]
supposely to speed up partclone in case the device where the read happens is
slowier than the device that is used for writes.
See mbuffer(1) manpage examples.
In any case, this needs benchmarking to really make sure this is helping.
Remove it until that ever happens.
cover more error cases where exceptions need to be raised.
check return code in the invoked subprocess.
restoreImageCustom has been intentionally left behind, it
is unclear what this custom script returns on success and
error.
validate 'Needed relocations: ' is in place before stepping on the split chunks
(2024-01-11 10:28:16) ogClient: [ERROR] - Exception when running "image create" subprocess
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 454, in image_create
ogReduceFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 105, in ogReduceFs
_reduce_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 235, in _reduce_ntfsresize
extra_size = int(out_resize_dryrun.split('Needed relocations : ')[1].split(' ')[0])*1.1+1024
IndexError: list index out of range
if not present, no need to adjust size
Users can create an image of a filesystem that contains no OS, therefore,
instead of rising an exception when no OS is detected, deliver a "unknown"
OS and an empty list of software.
Image backup is considered a legacy feature. Use the legacy mechanism of
naming image backups by adding ".ant" suffix.
Previously, by using the strftime suffix clients were reporting that the
disk were getting full rather quickly.
When a good method for image deletion is implemented then a proper
backup naming mechanism should be reconsidered.
When a client's hardware presents an empty pci storage child there is an
invalid call to _bytes_to_human a string is supplied as a default value
if the storage child does not present a 'size' attribute.
Fix this by checking if 'size' is present in the JSON output from lshw.
If size is present then map the bytes to a human readable string using
_bytes_to_human, if no size is present then use 'Empty slot' to indicate
that the memory bank is not being used.
When a client's hardware presents an empty memory bank and invalid call
to _bytes_to_human is performed because None is passed as a parameter.
size = _bytes_to_human(obj.get('size', None))
Fix this by checking if 'size' is present in the JSON output from lshw.
If size is present then map the bytes to a human readable string using
_bytes_to_human, if no size is present then use 'Empty slot' to indicate
that the memory bank is not being used.
Some users have mistakenly reported tiptorrent problems when the process
takes a long time. Specifically by rebooting or powering off the client
in the middle of the md5sum computation stage, just after the tiptorrent
transfer.
Same problem occurs when image creation command takes a long period of
time.
In order to help the user understand the different stages of commands
such as image creation or image restore using tiptorrent, the following
changes have been made to the current logging solution:
- Add log messages to warn users not to reboot or shut down the client
during a tiptorrent transfer, and also during the md5sum computation
stage.
- Add a log message telling the user that the image creation processes
have started.
- Use logging.exception inside "except:" blocks to print a traceback
with the log messsage.
(https://docs.python.org/3/library/logging.html#logging.exception)
The first stage of parsing the "lshw -json" command output is to load
the json string into a Python dictionary. lshw output is large and
varies from machine to machine, so it's not safe to assume that
different keys will be present in the dictionary.
Use dict.get() instead of dict[key] to avoid KeyError exceptions.
Backup image file if image creation request included
"backup": true
This only applies when the target image is already present in the
repository folder before running the partclone subprocess.
This parameter is ignored if the target image is not present in the
repository.
Enable parsing of "X-Sequence" HTTP headers from incoming requests.
Add "seq" field in restRequest class.
Enable adding "X-Sequence" to outgoing responses.
Add "seq" field inside restResponse class.
Store current client sequence number inside ogClient class.
Ideally, the restRequest object should be used to retrieve the
sequence number but not all processing functions inside ogRest.py
receive the request as parameter (eg: process_refresh).
In the other hand, all processing functions receive the ogClient object.
The subprocess module expects bytes-like object for "input" parameter by
default. Passing a string object result in the following error:
(2023-06-13 14:44:43) ogClient: [ERROR] - Exception when running "image create" subprocess
(2023-06-13 14:44:43) ogClient: [ERROR] - Unexpected error
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 465, in image_create
ogExtendFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 124, in ogExtendFs
_extend_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 250, in _extend_ntfsresize
proc = subprocess.run(cmd, input='y')
File "/usr/lib/python3.8/subprocess.py", line 495, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.8/subprocess.py", line 1013, in communicate
self._stdin_write(input)
File "/usr/lib/python3.8/subprocess.py", line 962, in _stdin_write
self.stdin.write(input)
TypeError: a bytes-like object is required, not 'str'
Fixes: dd999bfe34 ("utils: rewrite ogReduceFs")
There is a corner case in which a target NTFS filesystem is already
shrunken. When this happens ntfsresize text output parsing breaks.
Check when ntfsresize reports nothing to do, warn the user about this
and stop the dry-run ntfsresize loop.
_extend_ntfsresize contains an incorrect variable name inside
subprocess.run referring the resize command value.
Simplify this variable name inside each specific _extend_* function:
s/cmd_resize2fs/cmd
s/cmd_ntfsresize/cmd
Remove unnecessary InventarioSoftware invocation inside image_create
operation. Software inventory is executed after image creation
(see ogRest.py).
Remove legacy 'path' parameter. This parameter was used to specify the
path of a text file in which legacy bash scripts wrote the software
inventory of the client (something like "Csft-{ip}...").
Fixes: 04bb35bd86 ("live: rewrite software inventory")
Fixes: 2e3d47b7b8 ("Avoid writting /software output to a file")
Don't raise exception if any windows program is missing DisplayName
node in the windows registry.
This attribute/node should contain the program's name. This name is used
as the package's name in the software set (software inventory).
This patch should be considered a hotfix, python-hivex does not report
any helpful message about this error.
(2023-05-09 14:43:13) ogClient: [ERROR] - Unexpected error
Traceback (most recent call last):
[...]
RuntimeError: Success
Before this patch, image creation *might* fail because it cannot create
the software inventory associated with the image due to the previously
described error. The software inventory is part of the response payload
of the image creation command (see src/ogRest:image_create).
Fixes: 04bb35bd86 (live: rewrite software inventory)
Add optional 'operation' parameter to _poweroff_oglive function.
Reuse _poweroff_oglive code before the busybox subprocess when rebooting
an ogLive client.
Replace legacy bash script /opt/opengnsys/client/scripts/poweroff with a
Python native solution.
Use subprocess module for any required external program when shutting
down a client. ethtool is used to ensure WoL setting is correct before
shutting down.
ogLive does not properly use a init system so busybox is used when
shutting down the system. In other live environments poweroff operation
just calls /sbin/poweroff.
Add utility function to unmount any mountpoint present in the /mnt
folder.
This function is a simplified version of the legacy bash function
ogUnmountAll used in several operations.
Drop subprocess call to bash function ogExtendFs. Use a native python
solution with subprocess calls to the required underlying tools.
Use get_filesystem_type to get the present filesystem from a partition
and call the corresponding filesystem grow function.
Filesystem specific functions are declared "_extend_{filesystem}" and
should not be imported elsewhere.
Each filesystem specific function wraps a subprocess call to the
required underlying program:
- NTFS filesystems: "ntfsresize -f [partition]"
- ext4 filesystems: "resize2fs -f [partition]"
Set NTFS related subprocess stdin to 'y' because human input cannot be
unset with other ntfsresize parameters.
Jose M. Guisado
OpenGnSys Support Team
Alejandro Sirgo Rica