Revisit 5056b8f0d5 ("fs: validate ntfsresize dry-run output") that has
introduced a possible infinity loop.
Disentangle this loop while at it: iterate until best smallest size is
found by probing.
do not return the returncode, instead return an integer.
do not use
except CalledProcessError as e:
it causes a another exception while handling exception.
Remount the original image repository.
it should be possible to simplify this further by:
- stacking mounts, no need to umount initial repo and mount it again
when switching to the new repo, because remount back initial repo
might fail (!)
- use check=False and simply check for x.returncode
Remove mbuffer, this is never used.
mbuffer has been never been used since ogClient supports native image restore.
Originally this was used like this:
partclone [...] | mbuffer -q -M 40M | lzop [...]
supposely to speed up partclone in case the device where the read happens is
slowier than the device that is used for writes.
See mbuffer(1) manpage examples.
In any case, this needs benchmarking to really make sure this is helping.
Remove it until that ever happens.
cover more error cases where exceptions need to be raised.
check return code in the invoked subprocess.
restoreImageCustom has been intentionally left behind, it
is unclear what this custom script returns on success and
error.
validate 'Needed relocations: ' is in place before stepping on the split chunks
(2024-01-11 10:28:16) ogClient: [ERROR] - Exception when running "image create" subprocess
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 454, in image_create
ogReduceFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 105, in ogReduceFs
_reduce_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 235, in _reduce_ntfsresize
extra_size = int(out_resize_dryrun.split('Needed relocations : ')[1].split(' ')[0])*1.1+1024
IndexError: list index out of range
if not present, no need to adjust size
Users can create an image of a filesystem that contains no OS, therefore,
instead of rising an exception when no OS is detected, deliver a "unknown"
OS and an empty list of software.
Image backup is considered a legacy feature. Use the legacy mechanism of
naming image backups by adding ".ant" suffix.
Previously, by using the strftime suffix clients were reporting that the
disk were getting full rather quickly.
When a good method for image deletion is implemented then a proper
backup naming mechanism should be reconsidered.
When a client's hardware presents an empty pci storage child there is an
invalid call to _bytes_to_human a string is supplied as a default value
if the storage child does not present a 'size' attribute.
Fix this by checking if 'size' is present in the JSON output from lshw.
If size is present then map the bytes to a human readable string using
_bytes_to_human, if no size is present then use 'Empty slot' to indicate
that the memory bank is not being used.
When a client's hardware presents an empty memory bank and invalid call
to _bytes_to_human is performed because None is passed as a parameter.
size = _bytes_to_human(obj.get('size', None))
Fix this by checking if 'size' is present in the JSON output from lshw.
If size is present then map the bytes to a human readable string using
_bytes_to_human, if no size is present then use 'Empty slot' to indicate
that the memory bank is not being used.
Some users have mistakenly reported tiptorrent problems when the process
takes a long time. Specifically by rebooting or powering off the client
in the middle of the md5sum computation stage, just after the tiptorrent
transfer.
Same problem occurs when image creation command takes a long period of
time.
In order to help the user understand the different stages of commands
such as image creation or image restore using tiptorrent, the following
changes have been made to the current logging solution:
- Add log messages to warn users not to reboot or shut down the client
during a tiptorrent transfer, and also during the md5sum computation
stage.
- Add a log message telling the user that the image creation processes
have started.
- Use logging.exception inside "except:" blocks to print a traceback
with the log messsage.
(https://docs.python.org/3/library/logging.html#logging.exception)
The first stage of parsing the "lshw -json" command output is to load
the json string into a Python dictionary. lshw output is large and
varies from machine to machine, so it's not safe to assume that
different keys will be present in the dictionary.
Use dict.get() instead of dict[key] to avoid KeyError exceptions.
Backup image file if image creation request included
"backup": true
This only applies when the target image is already present in the
repository folder before running the partclone subprocess.
This parameter is ignored if the target image is not present in the
repository.
Enable parsing of "X-Sequence" HTTP headers from incoming requests.
Add "seq" field in restRequest class.
Enable adding "X-Sequence" to outgoing responses.
Add "seq" field inside restResponse class.
Store current client sequence number inside ogClient class.
Ideally, the restRequest object should be used to retrieve the
sequence number but not all processing functions inside ogRest.py
receive the request as parameter (eg: process_refresh).
In the other hand, all processing functions receive the ogClient object.
The subprocess module expects bytes-like object for "input" parameter by
default. Passing a string object result in the following error:
(2023-06-13 14:44:43) ogClient: [ERROR] - Exception when running "image create" subprocess
(2023-06-13 14:44:43) ogClient: [ERROR] - Unexpected error
Traceback (most recent call last):
File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 465, in image_create
ogExtendFs(disk, partition)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 124, in ogExtendFs
_extend_ntfsresize(partdev)
File "/opt/opengnsys/ogClient/src/utils/fs.py", line 250, in _extend_ntfsresize
proc = subprocess.run(cmd, input='y')
File "/usr/lib/python3.8/subprocess.py", line 495, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.8/subprocess.py", line 1013, in communicate
self._stdin_write(input)
File "/usr/lib/python3.8/subprocess.py", line 962, in _stdin_write
self.stdin.write(input)
TypeError: a bytes-like object is required, not 'str'
Fixes: dd999bfe34 ("utils: rewrite ogReduceFs")
There is a corner case in which a target NTFS filesystem is already
shrunken. When this happens ntfsresize text output parsing breaks.
Check when ntfsresize reports nothing to do, warn the user about this
and stop the dry-run ntfsresize loop.
_extend_ntfsresize contains an incorrect variable name inside
subprocess.run referring the resize command value.
Simplify this variable name inside each specific _extend_* function:
s/cmd_resize2fs/cmd
s/cmd_ntfsresize/cmd
Remove unnecessary InventarioSoftware invocation inside image_create
operation. Software inventory is executed after image creation
(see ogRest.py).
Remove legacy 'path' parameter. This parameter was used to specify the
path of a text file in which legacy bash scripts wrote the software
inventory of the client (something like "Csft-{ip}...").
Fixes: 04bb35bd86 ("live: rewrite software inventory")
Fixes: 2e3d47b7b8 ("Avoid writting /software output to a file")
Don't raise exception if any windows program is missing DisplayName
node in the windows registry.
This attribute/node should contain the program's name. This name is used
as the package's name in the software set (software inventory).
This patch should be considered a hotfix, python-hivex does not report
any helpful message about this error.
(2023-05-09 14:43:13) ogClient: [ERROR] - Unexpected error
Traceback (most recent call last):
[...]
RuntimeError: Success
Before this patch, image creation *might* fail because it cannot create
the software inventory associated with the image due to the previously
described error. The software inventory is part of the response payload
of the image creation command (see src/ogRest:image_create).
Fixes: 04bb35bd86 (live: rewrite software inventory)
Add optional 'operation' parameter to _poweroff_oglive function.
Reuse _poweroff_oglive code before the busybox subprocess when rebooting
an ogLive client.
Replace legacy bash script /opt/opengnsys/client/scripts/poweroff with a
Python native solution.
Use subprocess module for any required external program when shutting
down a client. ethtool is used to ensure WoL setting is correct before
shutting down.
ogLive does not properly use a init system so busybox is used when
shutting down the system. In other live environments poweroff operation
just calls /sbin/poweroff.
Add utility function to unmount any mountpoint present in the /mnt
folder.
This function is a simplified version of the legacy bash function
ogUnmountAll used in several operations.
Drop subprocess call to bash function ogExtendFs. Use a native python
solution with subprocess calls to the required underlying tools.
Use get_filesystem_type to get the present filesystem from a partition
and call the corresponding filesystem grow function.
Filesystem specific functions are declared "_extend_{filesystem}" and
should not be imported elsewhere.
Each filesystem specific function wraps a subprocess call to the
required underlying program:
- NTFS filesystems: "ntfsresize -f [partition]"
- ext4 filesystems: "resize2fs -f [partition]"
Set NTFS related subprocess stdin to 'y' because human input cannot be
unset with other ntfsresize parameters.
Drop subprocess call to bash function ogReduceFs. Use a native python
solution with subprocess calls to the required underlying tools.
Use get_filesystem_type to get the filesystem from a partition and call
the corresponding supported filesystem shrink function.
Filesystem specific functions are declared "_reduce_{filesystem}" and
should not be imported elsewhere.
In case of NTFS filesystems, the output of 'ntfsresize' is processed
directly. This is dirty, but we can expect no changes to the output
strings if we read the following comment in the nftsresize.c source
code:
https://github.com/tuxera/ntfs-3g/blob/edge/ntfsprogs/ntfsresize.c#L12
ntfsresize requires to do previous dry-run executions to confirm
that the resizing is possible.
If a dry-run fails but a 10% increase in size is still smaller than
original filesystem then retry the operation until dry-run reports
sucess or the size increase is bigger than original.
If resizing to a smaller ntfs filesystem is not possible then ogReduceFs
will do nothing.
Replace legacy shell script InventarioHardware for helper functions
from hw_inventory.py
Use get_hardware_inventory to obtain a HardwareInventory object with
the hardware information. Map the HardwareInventory object to a legacy
response string with the legacy_list_hardware_inventory function.
Remove "Chrd-*" file reading logic, it's no longer needed. Legacy shell
script InventarioHardware uses that file.
Expect a change in the structure of hardware inventory response payload
in the future. This patch does not address the HTTP response containing
the hardware inventory as a '\n' separated string of hardware elements.
hw_inventory.py defines classes and helpers functions enabling
fetching of hardware inventory from a running client.
Uses a subprocess call to the command 'lshw -json' to obtain hardware
information.
Relevant public functions:
> get_hardware_inventory()
Main function encapsulating subprocess and output processing
logic.
Returns a HardwareInventory object.
> legacy_list_hardware_inventory(inventory)
Legacy string representation of parameter HardwareInventory object
Rename software inventory file to sw_inventory to better distinguish
it from a future hardware inventory code.
In the future sw_inventory and hw_inventory might be merged together
once each file is tidied up.
Replace legacy bash script in favor of Python code. Improves error
traceability and further development.
The software inventory operation mounts the target partition and it
fetches the list of installed software (package set). Once the
operation is complete, it unmounts the target partition.
For Windows, introduce hivex library python bindings for accessing
Windows registry hive files (https://libguestfs.org/hivex.3.html).
This operation is still processed by legacy code in the server side
(ogAdmServer.c in ogServer). Legacy backend process expects the software
inventory like the following example:
"software": "Windows 10 Enterprise Evaluation 2004 \nIntel(R) Network Connections 24.0.0.11 24.0.0.11 ..."
The os name is inserted first in this list followed by a '\n' separated
string of the software packages.
The legacy server code can be found in function actualizaSoftware at
ogServer/src/ogAdmServer.c
It is expected for software inventory payload to change in the future to
a simpler solution using just a json array of strings.
Add missing samba credentials parameter in ogChangeRepo invocation.
Credentials are loaded from ogClient config file.
Any production deployment should use its own samba user and password.
ogChangeRepo fails when using default samba credentials in a production
environment.
Fixes: a1edbe904b ("legacy: rewrite ogChangeRepo")
Fixes: 3703fd6063 ("live: support native unicast cache image restore")
Removes undefined 'repo' variable from error logging message.
This caused the traceback to be polluted with an unhelpful message
about this variable being undefined.
Fixes: 3703fd606 ("live: support native unicast cache image restore")
Adds linux swap partition type, mapped to the 'LINUX-SWAP' string in web
interfaces like ogCP or webconsole.
Fixes: 29c53e54e9 ("live: add parttypes.py")
Capture all possible Python exceptions in the try/except block of every
opengnsys operation.
Create an error handling function to deduplicate code in the except
block. The error handling function resets the ogRest state to IDLE and
sends the corresponding 500 Internal Server Error.
This *does not cover* every possible error. There are functions inside
ogThread which contain code that may raise errors that are not covered
by any try/except block.
Remove unnecesary root logger constant: LOGGER
The root logger is used by default when executing:
logging.debug()
logging.info()
logging.warning()
...
There is no point in doing:
LOGGER = logging.getLogger() # Get root logger
LOGGER.debug() # Use root logger
Change the name of the helper functions used when getting opengnsys
image information (legacy ogGetImageInfo bash script). As of now the
process consist of decompressing the image file with lzop and feeding
that output to partclone.info.
Prefer a more explicit function name rather than "process_image_*"
Add comment about skipping the first two lines of partclone.info output.
Usually, partclone.info starts printing out these two lines that are not
related to the partclone image information:
Partclone v0.3.23 http://partclone.org
Showing info of image (-)
As long as partclone.info output doesn't change we'll be fine, but we
should not depend on human readable output. This might change in the
future (i.e. adding json output format to partclone.info).
Rewrites this legacy script behavior using native Python code, using
subprocess module when executing programs like partclone.info or lzop
ogGetImageInfo is a bash script that retrieves information regarding an
OpenGnsys partition image, specifically:
- clonator
- compressor
- filesystem
- datasize (size of the partition image)
This rewrite only supports partclone and lzop compressed images. This is
standard behavior, we have no reports of other programs or compression
algorithms in use.
Keep this legacy function with hungarian notation to emphasize this is
still a legacy component that may be replaced in the future.
Drop ogChangeRepo Bash script in favor of a native Python
approach. Use only necessary subprocess calls instead of bringing
all the logic of this function into a Bash script black box.
ogChangeRepo unmounts the current OpenGnsys image samba folder
(/opt/opengnsys/images) and mounts (connects to) a new directory using
the new provided ip address. Keeping access mode from previous mount.
If anything goes wrong when mounting the new directory, it will fallback
to mounting the previous directory.
If no previous OpenGnsys image samba directory is detected, this
functions tries to mount the new directory anyway. In this case,
it will raise CalledProcessError if something goes wrong.
Rewrites the setup operation using python-libfdisk module instead of an
external bash script. Consolidating the operation into Python's code,
limiting external subprocesses to well known programs and small
concrete tasks that are difficult to fully integrate into Python.
Use parttypes.py to fetch partition types from python-libfdisk module.
Use fs.py to create any specified supported filesystem.
OpenGnsys cache partitions are created labelling the partition as
"CACHE". Stops setting non-standard MBR hexcode (0xca) to the cache
partition in addition to the filesystem label.
Any partition specified as type EMPTY will be ignored.
init_cache() creates the default directory in which OpenGnsys stores
images when using any cache enabled transfer method.
As of this commit this folder must exist for tiptorrent.py to
work properly.
Subprocess Popen object inside tiptorrent.py use
'cwd' optional parameter like:
cwd='/opt/opengnsys/cache/opt/opengnsys/images/'
This folder convention might change in the future.
Adds utility module which wraps several mkfs.* calls as a subprocess.
The main utility function is mkfs(fs, disk, partition, label), which
subsequently calls the corresponding mkfs_*(partition_device) function.
mkfs() supports specifying a drive label where supported.
Other modules using fs.py should call mkfs() only.
Adds parttypes.py module with utility functions to get partition types
(parttypes) from python-libfdisk.
Supports standard partition types, either DOS or GPT.
DOS labels use a hex code to define partition types, python-libfdisk
exposes get_parttype_from_code to look up for DOS partition types from a
given hexcode.
GPT label uses a string (UUID) for each supported partition type,
python-libfdisk exposes get_parttype_from_string to look up for GPT
partition types from a given string.
Clients running in ogLive can show log messages via a lighttp server.
Particularly, a html page named "real time log" consists of <text-area>
tags with the contents of two particular text files
/tmp/session.log and /tmp/command.log
Adds a Python logging handler in order to write ogClient log messages
into /tmp/session.log. This way ogClient logs are show in the "real time
log" html page too.
Clears content of blue text areas in the real time log view before
executing a restore image operation.
Adds private function _ogbrowser_clear_logs, this function writes to a
couple of text files present in the ogLive environment.
The contents of this file are printed out to the blue text areas
in the "real time log" view.
Fix error paths in live operations which do not
reset the "browser" to the main page (one with the menu).
Add error logging messages when:
* _restartBrowser fails.
* ogChangeRepo fails.
Improve checksum fetch error handling. For example, when an invalid
repository IP is specified.
UNICAST-CACHE consist of:
1. Checking if the target image is already present at the opengnsys
cache partition. If so, check for integrity (local and remote
checksum). If the image is not present in the cache partition,
download the target image into it.
2. Restore the image from cache partition.
This commit add support for this operation natively from ogClient
Python's code.
Specifies the image path in the lzop subprocess string.
It might be interesting to study efficient mechanisms to read large
binary files in python before using open() with default parameters for
buffered binary reading.
Integrates image restore command into native ogClient code. Further
reduces the need for external Bash scripts.
After a succesful image restore, OS configuration is still using
external Bash script "osConfigure/osConfigureCustom".
ogCopyEfiBootloader is an invalid legacy bash function name.
Rename to the correct function name 'ogCopyEfiBootLoader' and
rename utility python wrapper too.
Fixes: 0bd037c1a409c65fbcb01355ee0dd6dca770330e
Do not return the subprocess result for ogReduceFs/ogExtendFs.
ogReduceFs works with or without the target filesystem mounted.
ogExtendFs requires the target filesystem to be mounted.
'ogMount' legacy script invocation should be replaced by a better
mount/umount wrapper.
Use legacy script that saves the Windows-specific content from the ESP
to the image target filesystem.
Current image restore solution from OpenGnsys scripts expect the EFI
partition to be stored in the target system partition. (Only for Windows
10)
For example, storing the ESP in the NTFS partition of a Windows image.
Expect use of bash script ogCopyEfiBootloader until further
integration is merged.
Integrates some parts of this operation into native code, eg: the md5
checksum computation.
Wraps non native processes and commands using the subprocess module.
For example, legacy.py stores bash commands pending integration.
Supports python >=3.6, expected until more modern ogLives are put into
production environments.
Enables details in libfidsk context constructor call. Ensures size are
displayed as bytes (and not as a human readable string).
Avoids further arithmetical errors (e.g: converting from bytes to KB)
parseGetConf was necessary for parsing output from OpenGnsys
script "getConfiguration", which has been replaced since commit
1ab981a539 ("live: replace getConfiguration with refresh")
Adds new logging handler redirecting messages to the log file
located in the Samba shared directory (applies to live mode
clients, i.e: ogLive)
Parses log level configuration from ogclient.json. See:
{
"opengnsys": {
...
"log": "INFO",
...
}
...
}
Adds --debug option to set root logger level to DEBUG when starting
ogClient. Overrides log level from config file.
In addition:
- Replaces any occurence of print with a corresponding logging function.
- Unsets log level for handlers, use root logger level instead.
- Default level for root logger is INFO.
- Replaces level from response log messages to debug (ogRest)
Old refresh was a wrapper around getConfiguration bash script.
New refresh operation does what getConfiguration used to do
externally. See commit 097769b971.
Returns true if target is already a mountpoint. Does not call mount.
It's possible that another device might be mounted in the target
mountpoint. A future check between the source and target for
equal device major:minor must be added.
Generates a cache.txt file if a cache partition is detected.
OpenGnsys stores information about stored images in its 'cache'
partition via a text file.
The file is stored in a samba shared directory, mounted at
'/opt/opengnsys/log/' in a live client. The file name is '{ip}.cache.txt'.
Previously, the generation of this file was delegated to external bash
scripts.
OpenGnsys partition images store OS information, including
the OS name appended with "64 bits" when the OS is meant for 64 bit
machines.
The detected OS name when probing (refresh) is important, if it differs
from what's stored in the DB OpenGnsys will wipe last image restored
information when running a refresh.
See actualizaConfiguracion from legacy ogserver (ogAdmServer.c) code:
dato = dbi_result_get_uint(result, "idnombreso");
if (idsoi == dato) {
swu = false;
}
...
if (swu) {
result_update = dbi_conn_queryf(dbi->conn,
"UPDATE ordenadores_particiones SET "
" codpar=0x%s,"
" tamano=%s,"
" uso=%s,"
" idsistemafichero=%d,"
" idnombreso=%d,"
" idimagen=0,"
" idperfilsoft=0,"
" fechadespliegue=NULL"
" WHERE idordenador=%d AND numdisk=%s AND numpar=%s",
Adds missing slash in the windows reghives path f-string.
When calling os_probe inside refresh (live/ogOperations) the mountpoint
is passed without a trailing slash.
Removes a workaround enumaration of the partitions of a given libfdisk
context. Use Partition class partno attribute.
This enables detecting non contiguous partitions, like:
/dev/sda
/dev/sda1
/dev/sda2
/dev/sda4
Use python binding for libfdisk to retrieve disk label and partition
information [1]. Use utils.py for the rest of the process.
This change aims to drop dependency with external script
'getConfiguration' from the OpenGnsys scripting ecosystem.
Keeps the same json payload structure for the refresh response.
[1] https://git.48k.eu/python-libfdisk/
partcodes.py is used for mapping GPT partition UIIDs to hexcode/decimal
values.
OpenGnsys stores partition types in the database using the decimal
value of this hexcode. The hexcode used for GPT partition UUID is the
same as defined internally in gdisk [1].
[1]
https://sourceforge.net/p/gptfdisk/code/ci/master/tree/parttypes.cc#l75
* disk.py
Disk discovery
* fs.py
Uses psutil to fetch fs usage information
* menu.py
ogBrowser menu generation
* net.py: gets nic status information
IP address, MAC address and ethernet speed.
* probe.py: probes mountpoints for operating systems
Uses hivexget command to try fetching Windows installation
information.
Looks for /etc/os-release for probing linux systems.
Commit 700aa89ea9 introduced the use of getattr when getting event
socket to avoid errors in case it was not initialized.
Prefer to initialize accordingly inside the ogClient class constructor.
If ogClient does not run on windows or linux mode, it does not create a
datagram event socket.
If ogClient runs in virtual or live mode this will cause an error when
calling get_event_socket because the class instance has no member
"event_sock".
Use getattr when retrieving the event socket from ogClient class, if
there is no event socket, then return None.
Fixes 2465ef25b7 (Add event datagram socket)
Running ogClient as a service (non interactive user) breaks
poweroff and reboot using ExitWindowsEx function in user32.dll.
Spawn a subshell using os.system and use the 'shutdown' command
instead.
This is a terminating command, we don't need fine grain from
subprocess module.
Commit 2dbcd18c06 breaks interfaceAdm path for live operations.
Keep OG_PATH to the value prior to commit 2dbcd18c06.
Check platform before reading config file. If platform is different
from linux then look for cfg/ogclient.json in current folder and
do not use OG_PATH.
ogClient can receive events via a datagram socket opened at 55885.
This socket is only opened when in windows or linux mode, for
event reporting from within the system.
Events reported this way are sent back to ogServer via a 103 Early
Hints HTTP message. Information regarding the event is sent in the
response's payload.
Windows does not report a refused connection the same way as Linux.
Unsuccesful connect socket will be kept in the exceptfds, and won't
be in the readable nor writable fds. The socket in this state will
have SO_ERROR set to ECONNREFUSED.
On the other hand, Linux does not use exceptfds for such case.
If current platform is Linux (either live, virtual or linux) expect
/opt/opengnsys/ogclient/cfg/ to contain ogclient.json.
If current platform is not Linux then we fallback to current directory.
Add agent mode for windows platform.
Subprocess module for shell/run is cross-platform an no change was
needed. The subprocess will run with the same privilege as its parent,
ogclient.
TODO: Provide a windows installer. As of now, an administrator
needs to install python and required libraries for this mode to be
usable.
We can't use syslog if we want to execute ogClient in the Windows
platform.
Use the native logging library so we can attach different handlers
depending on the mode ogClient is executing.
Logging configuration is done via a python dict. There is a different
dict for linux and windows. These dicts define the configuration of the
root logger, handlers and formatters used.
As of now, it is only expected to use the root logger for everything
logging related. The root logger is obtained via:
LOGGER = logging.getLogger()
More info about handlers, formatters and loggers:
https://docs.python.org/3/howto/logging.html
Logging configuration is done at startup, just after parsing the json
(knowing ogclient mode). If json parsing goes bad, ogclient will only
print a message to stdout.
- Executed script runs with same privilege as ogClient process.
- Uses subprocess.run instead of subprocess.Popen, it's a bit simpler.
We can't specify executable, though. Shouldn't need so in Linux mode.
- Uses shell=True, keep in mind security considerations listed at:
https://docs.python.org/3/library/subprocess.html#security-considerations
(shlex.quote can be used for unix shells)
Adds a systray icon for linux mode. Uses pystray module. Expects a
favicon.ico stored in the same folder as the main ogclient python
script, but if not found a placeholder image is used.
ogClient can run in "linux" mode. In addition to live or virtual.
Serves as a substitute to the legacy ogagent, which has not received any
updates since 2020/07/23.
Linux mode initially supports remote reboot and poweroff. Requires
updated ogServer with the Linux ogclient state.
ogClient can be set up to run in linux mode by specifying it in
ogclient.json:
{
"opengnsys": {
"ip": "192.168.56.10",
"port": 8889,
"log": "DEBUG",
"mode": "linux",
...
}
Abort command if the connection with the server is lost.
Otherwise, a race condition that leaves ogServer and ogClient
out-of-sync might occur:
1. ogClient is busy running a command (on the worker thread), for example,
image/restore.
2. ogServer is stopped OR ogClient loses connection with ogServer (due to
transient network problem).
3. ogClient reconnects and ogServer sends a refresh command.
4. ogClient worker thread finishes and it sends a reply to image/restore.
5. ogServer gets confused because it expects a reply to the refresh
command, not the old image/restore.
Polling for a qmp port availability is undesirable, as QEMU only handles
one connection to the qmp port at a time, ogClient may interfere with
cloneer-manager.
Check vm thread now connects to a separate qmp tcp socket, listening for
a shutdown guest event.
When ogClient is run just after ogVDI installation (before guest
installation) it will try to connect until it's possible, ie: after an
iso is specified and a qemu vm is started that exposes the appropiate
qmp tcp port.
If server closes the connection, close the socket, reconnect and stop
processing. self.connect() already cleans up the internal state, including the
socket state.
OgClient miscalculates the body size of the request.
ogServer delimits HTTP headers with "\r\n\r\n" to comply with RFC 2616.
But ogClient searches for the first "\r\n" delimiter, hence, ogClient
stops at the first HTTP header field instead of at the end of the header.
Hence, it incorrectly assumes the body starts after the first "\r\n".
This commit updates ogClient to search for the "\r\n\r\n" delimiter.
Example:
POST /shell/run HTTP/1.1\r\n <-- ogClient considers body starts here (WRONG!)
Content-Length: 952\r\n
Content-Type: application/json\r\n
\r\n <-- Here is where the body starts
{"json-body":...}
Add ogClient support to receive, parse and send disk type data from the cloning
engine when refreshing disks configuration.
See also commits with #1037 in ogServer and WebConsole repo.
Separates probe method into separate ogclient modes (virtual, vdi) so
future supported OS can easily have a tailored probe responses.
Link speed is retrieved using a minimal ethtool command sent using fcntl
module from python.
When running image_create operation an underlying script "imageCreate"
is called using the subprocess python module. This script may fail, for
example if the repository in which the image is to be stored has no
sufficent capacity.
Check for non-zero when the imageCreate process has finished.
ogLive related operations are named inside a 'Linux' folder, also its
python class is named OgLinuxOperations. Rename every 'linux' occurrence
with live to further clarify this folder and operations.
- OgLinuxOperations -> OgLiveOperations
- src/linux/ -> src/live/
Fixes: 1377acee ('Rename 'linux' mode to 'live' mode')
ogClient /software gets a representation of the target resource’s state.
GET method is more appropriate than POST.
Change /software method from POST to GET.
Before this patch the ogRest would hang indifinitely in a BUSY state
when a bad request was received. Fix this by returning ogRest state to
IDLE once the corresponding bad request response has been sent.
This accounts for the following cases:
- Unknown GET action
- Unknown POST action
- Unknown HTTP verb
ogClient /setup in linux mode do not support to indicate which table
type the user want to use. It always supposes that the partition table
is MBR/MSDOS.
Add ogClient support to work with GPT tables. Add new field table type
to /setup linux mode that expects a string with "MSDOS" or "GPT".
Example old JSON:
{
"disk": "1",
"cache": "0",
"cache_size": "0",
"partition_setup": [...]
}
Example new JSON:
{
"type": "GPT",
"disk": "1",
"cache": "0",
"cache_size": "0",
"partition_setup": [...]
}
Previous commits updates ogClient and ogServer to support several disks
in Linux (ogLive) mode. This changes disk setup JSON format from an
object to an array. ogClient in virtual mode also need to satisfy this
new format, although it not support several disks.
Adapt ogClient virtual mode to satisfy new disk setup JSON format.
Old format:
"disk_setup": {
"disk": 1,
"partition": 0,
"code": "0",
"filesystem": "",
"os": "",
"size": 32685957,
"used_size": 6
},
New format:
"disk_setup": [
{
"disk": 1,
"partition": 0,
"code": "0",
"filesystem": "",
"os": "",
"size": 32685957,
"used_size": 6
}
],
ogClient expect to receive information of 1 disk from the Cloning
Engine's script "getConfiguration". ogAdmClient, the deprecated ogLive
client daemon, supported for several disk.
Add ogClient support for several disk.
Image datasize is expressed in kibibytes but the existing REST API field
represent data in bytes.
This commit changes ogClient to send datasize in bytes.
Extend ogClient to include more information about the image that has been
created. This patch modifies ogClient to read an info file created by image
creation script, add this info to the JSON response and then remove the file.
Example of new /image/create response:
{
"disk": "1",
"partition": "1",
"code": "131",
"id": "1",
"name": "ubuntu",
"repository": "192.168.56.10",
"software": "Ubuntu 18.04.5 LTS \naccountsservice 0.6.45\n...",
"clonator": "PARTCLONE",
"compressor": "LZOP",
"filesystem": "EXTFS",
"datasize": 2100000
}
New fields are "clonator", "compressor", "filesystem" and "datasize".
Irina reports that "Partition assistant"/"Asistente de particionado" is
not working. This is happening because ogClient is not reading the full
data ogServer sends when the entire HTTP PDU is larger than 1024.
However, ogClient should read the whole message, reading until read data
length is greater or equal to "Content-Length" header value.
ogClient fails to obtain "Content-Length" value because is looking for
"content-length", be aware of the case sensitivity. It also needs to
take into account the header length because read data length also
includes headers.
This patch updates ogClient to:
1) look for "Content-Length instead of "content-length".
2) compare read date length with content length plus headers
length.
Disconnect gracefully from ogServer after booting OS, the script to start the
OS calls kexec, so everything is gone after it. For Windows, this results in a
reboot.
This commit also improves the disconnect function to make sure the
disconnection is synchronous.
This is useful for debuging and getting information on the processes that are
being executed in ogclient.
syslog outputs are something similar to:
Jun 26 10:36:40 ogAdministrator /ogclient: GET refresh HTTP/1.1
Jun 26 10:36:40 ogAdministrator /ogclient: HTTP/1.0 500 Internal Server Err
When restore image command was sent and the restoration failed, you
could see in WebConsole the image as restored and the command completed,
as it if had not failed. This happened because ogClient did not check
the return code of restoration script.
This commit adds return code check on restore image. So, when return
code is a non-zero value ogClient responses with an error 500. When
ogServer receives this error response, it did not set in the database
the image as restored and command as completed without errors.
Improves code encapsulation by moving check_vm_state_loop method into
OgVirtualOperations class. This also fixes import error when running ogclient in
'linux' mode.
Commit 621fb7a7 added class initialization with two parameters. These
two parameters were wrongly initialized, "self." was missing. The call
of these params in other functions of the class was also wrong.
This commit fix the initialization of the parameters of
OgLinuxOperations and fix the calls to these params.
ogClient may runs on an OS that do not have all the dependencies needed
to use virtual functions.
This commit change the behaviour to only import virtual functions when
the ogClient has to work with Virtual Machines. This way ogClient works
on environments which do not need virtual functions.
Ths patch handles different cases when a /refresh request can be made. More
specifically this handles the situation in which the VM is stopped when /refresh
is called but it starts during the processing of /refresh, in this case we just
send the last recorded setup.
This patch:
- Fixes logic errors in the communication with QMP (the order of handshake
messages was not right).
- Rewrite parts of OgQMP class.
- Enforces better coding practices by using Python's "context managers" to avoid
forgeting an open socket in case exceptions occur.
- Adapt virtual operations to the use of "context managers" using the "with"
statement.
This is the device that has the best empirical performance right now and it
seems like it is where most of the qemu development is directed towards for the
future.
ogClient changed its config file format to json. This patch adapts
ogLinuxOperation to use new config file.
Co-authored-by: Roberto Hueso <rhueso@soleta.eu>
This patch waits before polling qemu with QMP so that it has time to be ready
for requests. It also increases wait time in case the host machine is slow.
Write permission is not needed to get information about each virtual drive so
this patch changes permissions to 'only read'. This can prevent race conditions.
This patch calls poweroff in virtual mode when no VM is running and no jobs are
being executed. This is useful when the guest OS shutdowns so that the host OS
does not continue to run.
This patch includes changes to:
- Poweroff VM before running operations that require access to virtual disks.
- Poweroff VM before host system poweroff.
When virtual mode is activated, send the last stored partitions setup because
it is not possible to access a running virtual drive to get its information.
recv method is useful for receiving information that was not previously
requested (such as "events"). This patch also implements automatic handshake
on OgQMP by sending an "qmp_capabilities" request.
We observed that with the new ogClient the ogLive did not show the
realtime log menu*.
This commit changes the ogClient to launch the browser and show the
realtime log menu when some commands are executed.
* The realtime log menu is a menu that shows current status and info
about the command that the ogLive is executing.
We have observed that the ogLive browser did not show the changes when
an image restoration o script execution was called. The problem was that
the HTML menu was not updated/regenerated in these cases.
This commit adds the call to the HTML regeneration function to image
restore and script execution commands. This way, when the browser is
restarted it loads the updated HTML menu.
We have observed that the browser did not show changes. This happens
because the ogClient opens the browser at the beginning and forget
about it. To show correctly the changes ogClient had to restart the
browser every time the menu html changes.
This commit changes the behaviour to restart the browser when the menu
html has changed.
Testing the ogClient I found that if a value of the json match a key the
ogClient has an exception. For example:
body = "... shell/run {"run": "fdisk -l"} ..."
CURRENT
Enters in
if "disk" in body:...
if "run" in body:...
EXPECTED
Enters in
if "run" in body:...
This commit changes the behaviour to search for the keys in the
dictionary returned by json.loads() instead of searching in the raw
string. This way the ogClient looks for the keys without searching in
the values.
With our client disconnection, we hid that the ogClient process will be
closed too. This new way only close the subprocess keeping the ogClient
still working.
This patch allows us to control the thread using a state machine. This state
machine controls if the client is busy or not. This new way to control the
threads give us more control for avoiding race condition problems.
From pep-0008: Method Names and Instance Variables
Use the function naming rules: lowercase with words separated by underscores as
necessary to improve readability.
This is applied to /software and /hardware file reads.
We should use 'with' instead of opening and closing a file since this prevents
that files stay open after an exception is raised.
With this new patch, the image/create command will create a response message
with more information for the server. The new format is:
{"disk" : "1", "partition" : "1", "code" : "1", "id" : "1", "name" : "test",
"repository" : "192.168.2.4", "software" : "xyz"}
"xyz" will be the output saved during the execution of InventarioSoftware in
a specific path.
This patch allows us to execute refresh command using ogClient. This command
gets all the configuration in our machine and send this information to the
server. The format of the message that ogClient will send to the server will be:
{"disk": "1", "partition_setup": [{"partition": "1", "code": "LINUX",
"filesystem": "NTFS", "size": "498688", "format": "0"}, {"partition": "2",
"code": "LINUX", "filesystem": "NTFS", "size": "498688", "format": "0"},
{"partition": "3", "code": "LINUX", "filesystem": "NTFS", "size": "498688",
"format": "0"}]}
This patch allows us to use a new support for creating images using ogClient.
ogClient receives from the server a message which json body must be:
{"disk" : "1", "partition" : "1", "code" : "1", "id" : "1", "name" : "test",
"repository" : "192.168.2.4" }
ogClient returns to the server the software inventory executed before
create the image. The message for the server is:
{ "disk" : "0", "partition" : "1", "software" : "xyz" }
"xyz" will be the output saved during the execution of InventarioSoftware in
a specific path.
This patch adds a new echo option in /shell/run command. In case that the option
is set up to true, the server will receive in the response a json with the shell
output. Otherwise, the server will receive a response message without json
body.
A side effect of this change is that the command /shell/output/ disapears.
This patch adds a new response support. This way allows us to send a new
response message with more information about the partitions already set up.
The format of the response is:
{ "disk" : "1", "cache" : "1", "cache_size" : "0",
"partition_setup": [{"partition": "1", "code": "LINUX", "filesystem": "EMPTY",
"size": "498688", "format": "0"}...]
This patch allows us to send feedback to the server in case of error during
the execution of the command. In case of error, ogClient will send an
"Internal Error" http message.
This patch give us a better support in case of error or success execution.
In error cases, the new behavior is to send an Internal Error http message (500).
Otherwise, the server will receive a message with a json with this format:
{ "hardware" : "xyz" }
"xyz" is the output saved in a specific path during the execution of
InventarioHardware.
During our tests, we found some limitation during the execution of the software
command. We don't manage errors during the execution of this command. Moreover,
the server needs some information in case that everything is OK.
This patch modified the code for controlling the errors during the execution,
returning an "Internal Error" http message (500). Moreover, in case that
everything is OK, ogClient sends a message with this json body:
{ "disk" : "0", "partition" : "1", "software" : "xyz" }
"xyz" will be the output saved during the execution of InventarioSoftware in
a specific path.
This patch allows us to catch an exception when something wrong is happening
executing the init session command. In error cases, ogClient sends an internal
error http message to the server. Otherwise, an OK http message.
Now, all the arguments are received from httpparser. Those arguments convert
the function in long lines of codes. Passing directly the httpparser, all the
function will have less arguments and will be more clear the code.
ogAdmClient has a support to restore image on the machine. This new command
allows the new ogClient to execute the same script to restore the images
on the machine. The json format sent from the server must be:
{ "disk" : "1", "partition" : "1", "name" : "test",\
"repository" : "192.168.56.10", "type" : "UNICAST", "profile": "1", "id": "1"}
ogAdmClient has a support to configure the machines. This new command allows
the new ogClient to execute the same script to configure the machine.
The json format sent from the server must be:
{ "disk" : "1", "cache" : "0", "cache_size" : "70000000",\
"partition_setup": [{"partition": "1", "code": "NTFS", "filesystem": "NTFS",\
"size": "11000000", "format": "0"}]}
ogAdmClient has a support to inventory the hardware in a machine. This new
command allows the new ogClient to execute the same script to inventory the hw.
No arguments needed for executing the command
ogAdmClient has a support to inventory the software in a machine. This new
command allows the new ogClient to execute the same script to inventory the sw.
The arguments will be received from the server as a json message. Format:
{ "disk" : "0", "partition" : "1"}
ogAdmClient has a support for initializing the session in the machine. This new
command allows the new ogClient to execute the same script to init the session.
The arguments will be received from the server as a json message. Format:
{ "disk" : "0", "partition" : "1"}
This patch adds support for handling the error when the command is wrong formed.
Now, if we send a shell/run command and the shell command is incomplete,
the program crashes.
This patch changes the command process blocking the execution until is processing.
Moreover, the response will be OK (200) instead of IN_PROGRESS (202).
Right now, all the thread functions are declared inside the processor function.
Those functions were created for execute specific commands in the machine
(poweroff, reboot, etc). Creating this new class we are cleaning up the code.
Opengnsys needs a support to execute commands on the machine. This patch adds
the support for executing two new commands "shell/run" and "shell/output". The
first one, give us the support for executing a command in the machine and keep
save in a queue the output. The second one, give us the support for sending the
output from the command executed.
To disconnect the socket or send a message we use in different parts of the code
the attribute socket which is declared inside of the client. This way will
create us important problem in the future if we want to change the behavior in
our client sockets. This patch adds two new methods inside the client and we
can use them in other classes giving us the easy way to send messages and
disconnect the client.
This variable allow us to modify the opengnsys path without modify several parts
of the code. This change reduces the probability to add any bug forgetting to
change any line.
OpenGnSys Support Team
Alejandro Sirgo Rica
Jose M. Guisado
Javier Sánchez Parra
Roberto Hueso Gómez
Javier Sánchez Parra
Roberto Hueso Gomez
Alvaro Neira Ayuso