Update live shell run mode for the new REST API interface.
Evaluate the "inline" field to diferentiate between execution of
script in /opt/opengnsys/shell/ and a cmd execution.
Remove usage of echo argument of the API REST.
Update Windows and Linux mode for direct command execution.
Set OutputEncoding environment variable to 'utf-8' in Windows to
unify the encoding of stdout for the invoked programs.
Decode stdout to utf-8-sig to remove potential BOM.
While at this, remove strange legacy ;|\n\r terminator.
Detect user login and logout for Linux and Windows.
Report an active interactive session through the /refresh response
so a new ogserver instance can update the session status.
Poll the session change in 5 second intervals in a thread. Use the
same event socket previously used by the old session detection
mechanism to notify a session change.
Use the method check_interactive_session_change in each
ogOperations.py to report the session status.
Return values:
None: no session changes are found
True: login
False: logout
Windows
Verify if psutil.users() has any value.
Linux
Verify all the psutil.users() asociated to a terminal.
provide return code as result to ogserver.
Update virtual mode driver to return dummy value, although this command
is unimplemented, this seems to be broken due to possible TypeError when
accessing result from caller.
Add a 'cache' field into the json payload the client sends to
the server after a restore operation so the server can update
the new cache contents.
Resquest response structure:
{
...
'cache': [
{'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'},
{'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'}
]
...
}
Add API REST method to delete cache contents.
Resquest payload structure:
{
'images': ['windows.img', 'linux.img']
}
The client will try to delete as many images in cache as available
with names matching the list of filenames in the 'images' field.
Resquest response structure:
{
'cache': [
{'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'},
{'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'}
]
}
Log an error message in known error cases and log a backtrace
otherwise.
Define a new error type OgError to be used in all the 'raise'
blocks to define the error message to log. The exception
propagates until it reaches send_internal_server_error() where
the exception type is checked. If the type is OgError we log
the exception message. Logs the backtrace for other types.
The initial error implementation printed a backtrace everytime
an error ocurred. The next iteration changed it to only print
a backtrace in a very particular case but ended up omiting too
much information such as syntax errors or unknown error context.
The actual implementation only logs the cases we already cover in
the codebase and logs a bracktrace in the others, enabling a
better debugging experience.
Provide more information in exception messages as those are the
source of the logging messages. Add information about paths, files
or configuration related to the operation associated to the
exception.
Use only the exception messages as the main resource for error
messages.
The previous error code had string duplication in the form of:
logging.error('msg here')
raise Exception('msg here')
That approach also has the downside of having log duplication as
it had the local logging.err() and a global logging.exception()
inside send_internal_server_error capturing the exception message.
The actual code only requires raising an exception with a proper
error message.
Improve exception messages to give more error context.
Log every AssertionError as a backtrace.
Use the 'raise Exception from e' syntax to modify the a previously
raised exception 'e' into an exception with aditional context or
different type. This also prevents the message that warns about
newer exceptions being launch after an initial exception.
Enable parsing of "X-Sequence" HTTP headers from incoming requests.
Add "seq" field in restRequest class.
Enable adding "X-Sequence" to outgoing responses.
Add "seq" field inside restResponse class.
Store current client sequence number inside ogClient class.
Ideally, the restRequest object should be used to retrieve the
sequence number but not all processing functions inside ogRest.py
receive the request as parameter (eg: process_refresh).
In the other hand, all processing functions receive the ogClient object.
Remove unnecessary InventarioSoftware invocation inside image_create
operation. Software inventory is executed after image creation
(see ogRest.py).
Remove legacy 'path' parameter. This parameter was used to specify the
path of a text file in which legacy bash scripts wrote the software
inventory of the client (something like "Csft-{ip}...").
Fixes: 04bb35bd86 ("live: rewrite software inventory")
Fixes: 2e3d47b7b8 ("Avoid writting /software output to a file")
Replace legacy shell script InventarioHardware for helper functions
from hw_inventory.py
Use get_hardware_inventory to obtain a HardwareInventory object with
the hardware information. Map the HardwareInventory object to a legacy
response string with the legacy_list_hardware_inventory function.
Remove "Chrd-*" file reading logic, it's no longer needed. Legacy shell
script InventarioHardware uses that file.
Expect a change in the structure of hardware inventory response payload
in the future. This patch does not address the HTTP response containing
the hardware inventory as a '\n' separated string of hardware elements.
Capture all possible Python exceptions in the try/except block of every
opengnsys operation.
Create an error handling function to deduplicate code in the except
block. The error handling function resets the ogRest state to IDLE and
sends the corresponding 500 Internal Server Error.
This *does not cover* every possible error. There are functions inside
ogThread which contain code that may raise errors that are not covered
by any try/except block.
Remove unnecesary root logger constant: LOGGER
The root logger is used by default when executing:
logging.debug()
logging.info()
logging.warning()
...
There is no point in doing:
LOGGER = logging.getLogger() # Get root logger
LOGGER.debug() # Use root logger
Rewrites this legacy script behavior using native Python code, using
subprocess module when executing programs like partclone.info or lzop
ogGetImageInfo is a bash script that retrieves information regarding an
OpenGnsys partition image, specifically:
- clonator
- compressor
- filesystem
- datasize (size of the partition image)
This rewrite only supports partclone and lzop compressed images. This is
standard behavior, we have no reports of other programs or compression
algorithms in use.
Keep this legacy function with hungarian notation to emphasize this is
still a legacy component that may be replaced in the future.
Adds new logging handler redirecting messages to the log file
located in the Samba shared directory (applies to live mode
clients, i.e: ogLive)
Parses log level configuration from ogclient.json. See:
{
"opengnsys": {
...
"log": "INFO",
...
}
...
}
Adds --debug option to set root logger level to DEBUG when starting
ogClient. Overrides log level from config file.
In addition:
- Replaces any occurence of print with a corresponding logging function.
- Unsets log level for handlers, use root logger level instead.
- Default level for root logger is INFO.
- Replaces level from response log messages to debug (ogRest)
ogClient can receive events via a datagram socket opened at 55885.
This socket is only opened when in windows or linux mode, for
event reporting from within the system.
Events reported this way are sent back to ogServer via a 103 Early
Hints HTTP message. Information regarding the event is sent in the
response's payload.
Add agent mode for windows platform.
Subprocess module for shell/run is cross-platform an no change was
needed. The subprocess will run with the same privilege as its parent,
ogclient.
TODO: Provide a windows installer. As of now, an administrator
needs to install python and required libraries for this mode to be
usable.
We can't use syslog if we want to execute ogClient in the Windows
platform.
Use the native logging library so we can attach different handlers
depending on the mode ogClient is executing.
Logging configuration is done via a python dict. There is a different
dict for linux and windows. These dicts define the configuration of the
root logger, handlers and formatters used.
As of now, it is only expected to use the root logger for everything
logging related. The root logger is obtained via:
LOGGER = logging.getLogger()
More info about handlers, formatters and loggers:
https://docs.python.org/3/howto/logging.html
Logging configuration is done at startup, just after parsing the json
(knowing ogclient mode). If json parsing goes bad, ogclient will only
print a message to stdout.
ogClient can run in "linux" mode. In addition to live or virtual.
Serves as a substitute to the legacy ogagent, which has not received any
updates since 2020/07/23.
Linux mode initially supports remote reboot and poweroff. Requires
updated ogServer with the Linux ogclient state.
ogClient can be set up to run in linux mode by specifying it in
ogclient.json:
{
"opengnsys": {
"ip": "192.168.56.10",
"port": 8889,
"log": "DEBUG",
"mode": "linux",
...
}
Separates probe method into separate ogclient modes (virtual, vdi) so
future supported OS can easily have a tailored probe responses.
Link speed is retrieved using a minimal ethtool command sent using fcntl
module from python.
ogLive related operations are named inside a 'Linux' folder, also its
python class is named OgLinuxOperations. Rename every 'linux' occurrence
with live to further clarify this folder and operations.
- OgLinuxOperations -> OgLiveOperations
- src/linux/ -> src/live/
Fixes: 1377acee ('Rename 'linux' mode to 'live' mode')
ogClient /software gets a representation of the target resource’s state.
GET method is more appropriate than POST.
Change /software method from POST to GET.
Before this patch the ogRest would hang indifinitely in a BUSY state
when a bad request was received. Fix this by returning ogRest state to
IDLE once the corresponding bad request response has been sent.
This accounts for the following cases:
- Unknown GET action
- Unknown POST action
- Unknown HTTP verb
Image datasize is expressed in kibibytes but the existing REST API field
represent data in bytes.
This commit changes ogClient to send datasize in bytes.
Extend ogClient to include more information about the image that has been
created. This patch modifies ogClient to read an info file created by image
creation script, add this info to the JSON response and then remove the file.
Example of new /image/create response:
{
"disk": "1",
"partition": "1",
"code": "131",
"id": "1",
"name": "ubuntu",
"repository": "192.168.56.10",
"software": "Ubuntu 18.04.5 LTS \naccountsservice 0.6.45\n...",
"clonator": "PARTCLONE",
"compressor": "LZOP",
"filesystem": "EXTFS",
"datasize": 2100000
}
New fields are "clonator", "compressor", "filesystem" and "datasize".
Disconnect gracefully from ogServer after booting OS, the script to start the
OS calls kexec, so everything is gone after it. For Windows, this results in a
reboot.
This commit also improves the disconnect function to make sure the
disconnection is synchronous.
This is useful for debuging and getting information on the processes that are
being executed in ogclient.
syslog outputs are something similar to:
Jun 26 10:36:40 ogAdministrator /ogclient: GET refresh HTTP/1.1
Jun 26 10:36:40 ogAdministrator /ogclient: HTTP/1.0 500 Internal Server Err
Improves code encapsulation by moving check_vm_state_loop method into
OgVirtualOperations class. This also fixes import error when running ogclient in
'linux' mode.
ogClient may runs on an OS that do not have all the dependencies needed
to use virtual functions.
This commit change the behaviour to only import virtual functions when
the ogClient has to work with Virtual Machines. This way ogClient works
on environments which do not need virtual functions.
ogClient changed its config file format to json. This patch adapts
ogLinuxOperation to use new config file.
Co-authored-by: Roberto Hueso <rhueso@soleta.eu>
This patch waits before polling qemu with QMP so that it has time to be ready
for requests. It also increases wait time in case the host machine is slow.
This patch calls poweroff in virtual mode when no VM is running and no jobs are
being executed. This is useful when the guest OS shutdowns so that the host OS
does not continue to run.
Alejandro Sirgo Rica
OpenGnSys Support Team
Jose M. Guisado
Javier Sánchez Parra
Roberto Hueso Gómez