suzanne.soy/qubes-doc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'deathgrippin-automated-tests-patch'

Browse Source
This commit is contained in:
Andrew David Wong 2020-06-04 08:15:10 -05:00
commit 12992b6b79
No known key found for this signature in database
GPG Key ID: 8CE137352A019A17
1 changed files with 12 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
developer/debugging/automated-tests.md
View File

@ -22,10 +22,13 @@ Integration tests are written with the assumption that they will be called on de
Since these tests were written with this expectation, all the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing VM components. Since these tests were written with this expectation, all the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing VM components.
Most of the tests are stored in the [core-admin repository](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) in the `qubes/tests` directory. Most of the tests are stored in the [core-admin repository](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) in the `qubes/tests` directory.
To start them you can use standard python unittest runner: To start them you can use the standard python unittest runner:
python3 -m unittest -v qubes.tests
`python3 -m unittest -v qubes.tests`
Or our custom one: Or our custom one:
python3 -m qubes.tests.run -v
`python3 -m qubes.tests.run -v`
Our test runner runs mostly the same as the standard one, but it has some nice additional features like color output and not needing the "qubes.test" prefix. Our test runner runs mostly the same as the standard one, but it has some nice additional features like color output and not needing the "qubes.test" prefix.
It also has the ability to run lone selected template tests. It also has the ability to run lone selected template tests.
@ -101,7 +104,7 @@ Example test run:
### Qubes 4.0 ### Qubes 4.0
Tests on Qubes 4.0 require stopping `qubesd` service first, because special instance of it is started as part of the test run. Tests on Qubes 4.0 require stopping the `qubesd` service first, because a special instance of it is started as part of the test run.
Additionally, tests needs to be started as root. The full command to run the tests is: Additionally, tests needs to be started as root. The full command to run the tests is:
sudo systemctl stop qubesd; sudo -E python3 -m qubes.tests.run -v ; sudo systemctl start qubesd sudo systemctl stop qubesd; sudo -E python3 -m qubes.tests.run -v ; sudo systemctl start qubesd
@ -144,10 +147,10 @@ Again, given the hypothetical `example.py` test:
### Testing PyQt applications ### Testing PyQt applications
When testing (Py)QT application, it's useful to create separate QApplication object for each test. When testing (Py)QT applications, it's useful to create a separate QApplication object for each test.
But QT framework does not allow to have multiple QApplication objects in the same process at the same time. But QT framework does not allow multiple QApplication objects in the same process at the same time.
This means it's critical to reliably cleanup previous instance before creating the new one. This means it's critical to reliably cleanup the previous instance before creating a new one.
This turns out to be non-trivial task, especially if _any_ test uses event loop. This turns out to be a non-trivial task, especially if _any_ test uses the event loop.
Failure to perform proper cleanup in many cases results in SEGV. Failure to perform proper cleanup in many cases results in SEGV.
Below you can find steps for the proper cleanup: Below you can find steps for the proper cleanup:
@ -205,7 +208,7 @@ Installation Tests with openQA
Manually testing the installation of Qubes OS is a time-consuming process. Manually testing the installation of Qubes OS is a time-consuming process.
We use [openQA] to automate this process. We use [openQA] to automate this process.
It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses. It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses.
Then, it checks the output to see whether various tests were passed, e.g., by comparing the virtual screen output to screenshots of a successful installation. Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation.
Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT. Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT.
KVM also supports nested virtualization, so HVM should theoretically work. KVM also supports nested virtualization, so HVM should theoretically work.