Mount the system partition in readonly mode and check for the
hiberfil.sys file if the target system is a Windows.
Fail the image creation process if the target system is hibernated.
If formatting fails, log shows:
mkfs.ntfs reports return code 1 for /dev/sda2
but ogclient reports success to the ogserver.
Raise an exception so front-end gets an indication that formatting has failed.
Add exception checks to the os.mkdir operation and log the error
found. The previous implementation was too optimistic and only
handled mount related errors.
Report mkfs failure for every partition. This does not raise an
exception as that would skip partprobe operations and the mkfs
operations in the next potentially well formated partitions.
otherwise error path uses uninitialized variable
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 112, in ogReduceFs
return ret
UnboundLocalError: local variable 'ret' referenced before assignment
Log an error message in known error cases and log a backtrace
otherwise.
Define a new error type OgError to be used in all the 'raise'
blocks to define the error message to log. The exception
propagates until it reaches send_internal_server_error() where
the exception type is checked. If the type is OgError we log
the exception message. Logs the backtrace for other types.
The initial error implementation printed a backtrace everytime
an error ocurred. The next iteration changed it to only print
a backtrace in a very particular case but ended up omiting too
much information such as syntax errors or unknown error context.
The actual implementation only logs the cases we already cover in
the codebase and logs a bracktrace in the others, enabling a
better debugging experience.
Use only the exception messages as the main resource for error
messages.
The previous error code had string duplication in the form of:
logging.error('msg here')
raise Exception('msg here')
That approach also has the downside of having log duplication as
it had the local logging.err() and a global logging.exception()
inside send_internal_server_error capturing the exception message.
The actual code only requires raising an exception with a proper
error message.
Improve exception messages to give more error context.
Log every AssertionError as a backtrace.
Use the 'raise Exception from e' syntax to modify the a previously
raised exception 'e' into an exception with aditional context or
different type. This also prevents the message that warns about
newer exceptions being launch after an initial exception.
value extraction did not have error checking and was handled in
a one-liner. The actual implementation expands the parsing logic
and moves it into a function.
Revisit 5056b8f0d5 ("fs: validate ntfsresize dry-run output") that has
introduced a possible infinity loop.
Disentangle this loop while at it: iterate until best smallest size is
found by probing.
cover more error cases where exceptions need to be raised.
check return code in the invoked subprocess.
restoreImageCustom has been intentionally left behind, it
is unclear what this custom script returns on success and
error.
validate 'Needed relocations: ' is in place before stepping on the split chunks
(2024-01-11 10:28:16) ogClient: [ERROR] - Exception when running "image create" subprocess
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 454, in image_create
ogReduceFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 105, in ogReduceFs
_reduce_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 235, in _reduce_ntfsresize
extra_size = int(out_resize_dryrun.split('Needed relocations : ')[1].split(' ')[0])*1.1+1024
IndexError: list index out of range
if not present, no need to adjust size
The subprocess module expects bytes-like object for "input" parameter by
default. Passing a string object result in the following error:
(2023-06-13 14:44:43) ogClient: [ERROR] - Exception when running "image create" subprocess
(2023-06-13 14:44:43) ogClient: [ERROR] - Unexpected error
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 465, in image_create
ogExtendFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 124, in ogExtendFs
_extend_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 250, in _extend_ntfsresize
proc = subprocess.run(cmd, input='y')
File "/usr/lib/python3.8/subprocess.py", line 495, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.8/subprocess.py", line 1013, in communicate
self._stdin_write(input)
File "/usr/lib/python3.8/subprocess.py", line 962, in _stdin_write
self.stdin.write(input)
TypeError: a bytes-like object is required, not 'str'
Fixes: dd999bfe34 ("utils: rewrite ogReduceFs")
There is a corner case in which a target NTFS filesystem is already
shrunken. When this happens ntfsresize text output parsing breaks.
Check when ntfsresize reports nothing to do, warn the user about this
and stop the dry-run ntfsresize loop.
_extend_ntfsresize contains an incorrect variable name inside
subprocess.run referring the resize command value.
Simplify this variable name inside each specific _extend_* function:
s/cmd_resize2fs/cmd
s/cmd_ntfsresize/cmd
Add utility function to unmount any mountpoint present in the /mnt
folder.
This function is a simplified version of the legacy bash function
ogUnmountAll used in several operations.
Drop subprocess call to bash function ogExtendFs. Use a native python
solution with subprocess calls to the required underlying tools.
Use get_filesystem_type to get the present filesystem from a partition
and call the corresponding filesystem grow function.
Filesystem specific functions are declared "_extend_{filesystem}" and
should not be imported elsewhere.
Each filesystem specific function wraps a subprocess call to the
required underlying program:
- NTFS filesystems: "ntfsresize -f [partition]"
- ext4 filesystems: "resize2fs -f [partition]"
Set NTFS related subprocess stdin to 'y' because human input cannot be
unset with other ntfsresize parameters.
Drop subprocess call to bash function ogReduceFs. Use a native python
solution with subprocess calls to the required underlying tools.
Use get_filesystem_type to get the filesystem from a partition and call
the corresponding supported filesystem shrink function.
Filesystem specific functions are declared "_reduce_{filesystem}" and
should not be imported elsewhere.
In case of NTFS filesystems, the output of 'ntfsresize' is processed
directly. This is dirty, but we can expect no changes to the output
strings if we read the following comment in the nftsresize.c source
code:
https://github.com/tuxera/ntfs-3g/blob/edge/ntfsprogs/ntfsresize.c#L12
ntfsresize requires to do previous dry-run executions to confirm
that the resizing is possible.
If a dry-run fails but a 10% increase in size is still smaller than
original filesystem then retry the operation until dry-run reports
sucess or the size increase is bigger than original.
If resizing to a smaller ntfs filesystem is not possible then ogReduceFs
will do nothing.
Adds utility module which wraps several mkfs.* calls as a subprocess.
The main utility function is mkfs(fs, disk, partition, label), which
subsequently calls the corresponding mkfs_*(partition_device) function.
mkfs() supports specifying a drive label where supported.
Other modules using fs.py should call mkfs() only.
Do not return the subprocess result for ogReduceFs/ogExtendFs.
ogReduceFs works with or without the target filesystem mounted.
ogExtendFs requires the target filesystem to be mounted.
'ogMount' legacy script invocation should be replaced by a better
mount/umount wrapper.
Integrates some parts of this operation into native code, eg: the md5
checksum computation.
Wraps non native processes and commands using the subprocess module.
For example, legacy.py stores bash commands pending integration.
Supports python >=3.6, expected until more modern ogLives are put into
production environments.
Returns true if target is already a mountpoint. Does not call mount.
It's possible that another device might be mounted in the target
mountpoint. A future check between the source and target for
equal device major:minor must be added.
OpenGnsys partition images store OS information, including
the OS name appended with "64 bits" when the OS is meant for 64 bit
machines.
The detected OS name when probing (refresh) is important, if it differs
from what's stored in the DB OpenGnsys will wipe last image restored
information when running a refresh.
See actualizaConfiguracion from legacy ogserver (ogAdmServer.c) code:
dato = dbi_result_get_uint(result, "idnombreso");
if (idsoi == dato) {
swu = false;
}
...
if (swu) {
result_update = dbi_conn_queryf(dbi->conn,
"UPDATE ordenadores_particiones SET "
" codpar=0x%s,"
" tamano=%s,"
" uso=%s,"
" idsistemafichero=%d,"
" idnombreso=%d,"
" idimagen=0,"
" idperfilsoft=0,"
" fechadespliegue=NULL"
" WHERE idordenador=%d AND numdisk=%s AND numpar=%s",
* disk.py
Disk discovery
* fs.py
Uses psutil to fetch fs usage information
* menu.py
ogBrowser menu generation
* net.py: gets nic status information
IP address, MAC address and ethernet speed.
* probe.py: probes mountpoints for operating systems
Uses hivexget command to try fetching Windows installation
information.
Looks for /etc/os-release for probing linux systems.
Alejandro Sirgo Rica
OpenGnSys Support Team
Jose M. Guisado