Update assigning-devices.md
This commit is contained in:
parent
c87072d798
commit
2891ac92a1
|
@ -11,30 +11,19 @@ redirect_from:
|
||||||
Assigning Devices to VMs
|
Assigning Devices to VMs
|
||||||
========================
|
========================
|
||||||
|
|
||||||
Sometimes you may need to assign an entire PCI or PCI Express device directly
|
Sometimes you may need to assign an entire PCI or PCI Express device directly to a qube.
|
||||||
to a qube.
|
|
||||||
This is also known as PCI pass-through.
|
This is also known as PCI pass-through.
|
||||||
The Qubes installer does this by default for `sys-net` (assigning all network class controllers),
|
The Qubes installer does this by default for `sys-net` (assigning all network class controllers), as well as `sys-usb` (assigning all USB controllers) if you chose to create the USB qube during install.
|
||||||
as well as `sys-usb` (assigning all USB controllers) if you chose to create the
|
While this covers most use cases, there are some occasions when you may want to manually assign one NIC to `sys-net` and another to a custom NetVM, or have some other type of PCI controller you want to manually assign.
|
||||||
USB qube during install.
|
|
||||||
While this covers most use cases, there are some occasions when you may want to
|
|
||||||
manually assign one NIC to `sys-net` and another to a custom NetVM,
|
|
||||||
or have some other type of PCI controller you want to manually assign.
|
|
||||||
|
|
||||||
Note that one can only assign full PCI or PCI Express devices by default.
|
Note that one can only assign full PCI or PCI Express devices by default.
|
||||||
This limit is imposed by the PC and VT-d architectures.
|
This limit is imposed by the PC and VT-d architectures.
|
||||||
This means if a PCI device has multiple functions, all instances
|
This means if a PCI device has multiple functions, all instances of it need to be assigned to the same qube unless you have disabled the strict requirement for FLR with the `no-strict-reset` (R4.0) or `pci_strictreset` (R3.2) option.
|
||||||
of it need to be assigned to the same qube unless you have disabled FLR with the
|
In the steps below, you can tell if this is needed if you see the BDF for the same device listed multiple times with only the number after the "." changing.
|
||||||
`no-strict-reset` (R4.0) or `pci_strictreset` (R3.2) option.
|
|
||||||
In the steps below, you can tell if this is needed if you see the BDF for the
|
|
||||||
same device listed multiple times with only the number after the "." changing.
|
|
||||||
|
|
||||||
While a device can only be attached to one VM at a time, it *is* possible to
|
While PCI device can only be used by one powered on VM at a time, it *is* possible to *assign* the same device to more than one VM at a time.
|
||||||
*assign* the same device to more than one VM at a time.
|
This means that you can use the device in one VM, shut that VM down, start up a different VM (to which the same device is also assigned), then use the device in that VM.
|
||||||
This means that you can use the device in one VM, shut that VM down, start up a different VM
|
This can be useful if, for example, you have only one USB controller, but you have multiple security domains which all require the use of different USB devices.
|
||||||
(to which the same device is also assigned), then use the device in that VM.
|
|
||||||
This can be useful if, for example, you have only one USB controller, but you have multiple
|
|
||||||
security domains which all require the use of different USB devices.
|
|
||||||
|
|
||||||
R4.0
|
R4.0
|
||||||
------------------------
|
------------------------
|
||||||
|
@ -48,15 +37,13 @@ qvm-pci
|
||||||
|
|
||||||
This will show you the `backend:BDF` address of each PCI device.
|
This will show you the `backend:BDF` address of each PCI device.
|
||||||
It will look something like `dom0:00_1a.0`.
|
It will look something like `dom0:00_1a.0`.
|
||||||
Once you've found the address of the device you want to
|
Once you've found the address of the device you want to assign, then attach it like so:
|
||||||
assign, then attach it like so:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci attach --persistent <vmname> <backend>:<bdf>
|
qvm-pci attach --persistent <vmname> <backend>:<bdf>
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
For example, if `00_1a.0` is the BDF of the device you want to assign to the
|
For example, if `00_1a.0` is the BDF of the device you want to assign to the "personal" domain, you would do this:
|
||||||
"personal" domain, you would do this:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci attach --persistent personal dom0:00_1a.0
|
qvm-pci attach --persistent personal dom0:00_1a.0
|
||||||
|
@ -65,8 +52,8 @@ qvm-pci attach --persistent personal dom0:00_1a.0
|
||||||
R3.2
|
R3.2
|
||||||
------------------------
|
------------------------
|
||||||
|
|
||||||
In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci`
|
In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci` tool.
|
||||||
tool. First, list the available PCI devices:
|
First, list the available PCI devices:
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
lspci
|
lspci
|
||||||
|
@ -74,15 +61,13 @@ lspci
|
||||||
|
|
||||||
This will show you the BDF address of each PCI device.
|
This will show you the BDF address of each PCI device.
|
||||||
It will look something like `00:1a.0`.
|
It will look something like `00:1a.0`.
|
||||||
Once you've found the BDF address of the device you want to
|
Once you've found the BDF address of the device you want to assign, then attach it like so:
|
||||||
assign, then attach it like so:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci -a <vmname> <bdf>
|
qvm-pci -a <vmname> <bdf>
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
For example, if `00:1a.0` is the BDF of the device you want to assign to the
|
For example, if `00:1a.0` is the BDF of the device you want to assign to the "personal" domain, you would do this:
|
||||||
"personal" domain, you would do this:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci -a personal 00:1a.0
|
qvm-pci -a personal 00:1a.0
|
||||||
|
@ -100,16 +85,11 @@ Finding the right USB controller
|
||||||
--------------------------------
|
--------------------------------
|
||||||
|
|
||||||
Some USB devices are not compatible with the USB pass-through method Qubes employs.
|
Some USB devices are not compatible with the USB pass-through method Qubes employs.
|
||||||
In situations like this, you can still often get the USB device to work by
|
In situations like this, you can still often get the USB device to work by passing through the entire USB controller to a qube.
|
||||||
passing through the entire USB controller to a qube.
|
However, with this approach one cannot assign single USB devices, only the whole USB controller with whatever USB devices are connected to it.
|
||||||
However, with this approach one cannot assign single USB devices,
|
More information on using and managing USB devices with qubes is available on the [USB] page.
|
||||||
only the whole USB controller with whatever USB devices are connected to it.
|
If you want assign a certain USB device to a VM (by attaching the whole USB controller), you need to figure out which PCI device is the right controller.
|
||||||
More information on using and managing USB devices with qubes is
|
First, check to which USB bus the device is connected (note that these steps need to be run from a terminal inside `dom0`):
|
||||||
available on the [USB] page.
|
|
||||||
If you want assign a certain USB device to a VM (by attaching the whole USB controller),
|
|
||||||
you need to figure out which PCI device is the right controller.
|
|
||||||
First, check to which USB bus the device is connected (note that
|
|
||||||
these steps need to be run from a terminal inside `dom0`):
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
lsusb
|
lsusb
|
||||||
|
@ -124,8 +104,7 @@ Bus 003 Device 003: ID 413c:818d Dell Computer Corp.
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
The device is connected to USB bus \#3.
|
The device is connected to USB bus \#3.
|
||||||
Then check which other devices are connected to the same bus,
|
Then check which other devices are connected to the same bus, since *all* of them will be assigned to the same VM.
|
||||||
since *all* of them will be assigned to the same VM.
|
|
||||||
Now is the time to find right USB controller:
|
Now is the time to find right USB controller:
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
|
@ -139,19 +118,16 @@ This should output something like:
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
Now you see the BDF address in the path (right before final `usb3`).
|
Now you see the BDF address in the path (right before final `usb3`).
|
||||||
Strip the leading `0000:` and pass the rest to the `qvm-pci` tool to attach the controller
|
Strip the leading `0000:` and pass the rest to the `qvm-pci` tool to attach the controller with the version specific steps above.
|
||||||
with the version specific steps above.
|
|
||||||
|
|
||||||
Possible issues
|
Possible issues
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
### DMA buffer size
|
### DMA buffer size
|
||||||
|
|
||||||
VMs with assigned PCI devices in Qubes have allocated a small buffer for DMA
|
VMs with assigned PCI devices in Qubes have allocated a small buffer for DMA operations (called swiotlb).
|
||||||
operations (called swiotlb).
|
|
||||||
By default it is 2MB, but some devices need a larger buffer.
|
By default it is 2MB, but some devices need a larger buffer.
|
||||||
To change this allocation, edit VM's kernel parameters
|
To change this allocation, edit VM's kernel parameters (this is expressed in 512B chunks):
|
||||||
(this is expressed in 512B chunks):
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
# qvm-prefs netvm |grep kernelopts
|
# qvm-prefs netvm |grep kernelopts
|
||||||
|
@ -159,8 +135,7 @@ kernelopts : iommu=soft swiotlb=2048 (default)
|
||||||
# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=8192"
|
# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=8192"
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
This is [known to be needed][ml1] for the Realtek RTL8111DL Gigabit Ethernet
|
This is [known to be needed][ml1] for the Realtek RTL8111DL Gigabit Ethernet Controller.
|
||||||
Controller.
|
|
||||||
|
|
||||||
### PCI passthrough issues
|
### PCI passthrough issues
|
||||||
|
|
||||||
|
@ -168,22 +143,18 @@ Sometimes the PCI arbitrator is too strict.
|
||||||
There is a way to enable permissive mode for it.
|
There is a way to enable permissive mode for it.
|
||||||
See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page.
|
See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page.
|
||||||
|
|
||||||
**NOTE:** By setting the permissive flag for the PCI device, you're potentially
|
**NOTE:** By setting the permissive flag for the PCI device, you're potentially weakening the device isolation, especially if your system is not equipped with a VT-d Interrupt Remapping unit.
|
||||||
weakening the device isolation, especially if your system is not equipped with
|
|
||||||
VT-d Interrupt Remapping unit.
|
|
||||||
See [Software Attacks on Intel VT-d] (page 7)
|
See [Software Attacks on Intel VT-d] (page 7)
|
||||||
for more details.
|
for more details.
|
||||||
|
|
||||||
At other times, you may instead need to disable the FLR requirement on a device.
|
At other times, you may instead need to disable the FLR requirement on a device.
|
||||||
This will also weaken device isolation; see the "I created a usbVM..." entry in
|
This will also weaken device isolation; see the "I created a usbVM..." entry in the [FAQ](/doc/user-faq/) for more details.
|
||||||
the [FAQ](/doc/user-faq/) for more details.
|
|
||||||
|
|
||||||
R4.0
|
R4.0
|
||||||
------------------------
|
------------------------
|
||||||
|
|
||||||
Permissive mode and strict reset are options set as part of PCI device attachment.
|
Permissive mode and strict reset are options set as part of PCI device attachment.
|
||||||
If you've already attached the PCI device to a VM, detach it first either with Qube Manager
|
If you've already attached the PCI device to a VM, detach it first either with Qube Manager or `qvm-pci`, then list the available PCI devices:
|
||||||
or `qvm-pci`, then list the available PCI devices:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci
|
qvm-pci
|
||||||
|
@ -191,26 +162,22 @@ qvm-pci
|
||||||
|
|
||||||
This will show you the `backend:BDF` address of each PCI device.
|
This will show you the `backend:BDF` address of each PCI device.
|
||||||
It will look something like `dom0:00_1a.0`.
|
It will look something like `dom0:00_1a.0`.
|
||||||
Once you've found the address of the device you want to
|
Once you've found the address of the device you want to assign, then attach it like so:
|
||||||
assign, then attach it like so:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci attach --persistent --option <option1> [--option <option2>] <vmname> <backend>:<bdf>
|
qvm-pci attach --persistent --option <option1> [--option <option2>] <vmname> <backend>:<bdf>
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
For example, if `00_1a.0` is the BDF of the device you want to assign to the
|
For example, if `00_1a.0` is the BDF of the device you want to assign to the "personal" domain, and it is particularly difficult to pass through you would do this:
|
||||||
"personal" domain, and it is particularly difficult to pass through you would do this:
|
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true personal dom0:00_1a.0
|
qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true personal dom0:00_1a.0
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
Running `qvm-pci` again should then show your PCI device attached with both the
|
Running `qvm-pci` again should then show your PCI device attached with both the `permissive` and `no-strict-reset` options set.
|
||||||
`permissive` and `no-strict-reset` options set.
|
|
||||||
|
|
||||||
**Note** again that in most cases you should not need either of these options set.
|
**Note** again that in most cases you should not need either of these options set.
|
||||||
Only set one or more of them as required to get your device to function,
|
Only set one or more of them as required to get your device to function, or replace the device with one that functions properly with Qubes.
|
||||||
or replace the device with one that functions properly with Qubes.
|
|
||||||
|
|
||||||
R3.2
|
R3.2
|
||||||
------------------------
|
------------------------
|
||||||
|
@ -242,17 +209,14 @@ qvm-prefs usbVM -s pci_strictreset false
|
||||||
```
|
```
|
||||||
|
|
||||||
**Note** again that in most cases you should not need either of these options set.
|
**Note** again that in most cases you should not need either of these options set.
|
||||||
Only set one or more of them as required to get your device to function,
|
Only set one or more of them as required to get your device to function, or replace the device with one that functions properly with Qubes.
|
||||||
or replace the device with one that functions properly with Qubes.
|
|
||||||
|
|
||||||
Bringing PCI device back to dom0
|
Bringing PCI device back to dom0
|
||||||
--------------------------------
|
--------------------------------
|
||||||
|
|
||||||
By default, when a device is detached from a VM (or when a VM with an attached
|
By default, when a device is detached from a VM (or when a VM with an attached PCI device is shut down), the device is *not* automatically attached back to dom0.
|
||||||
PCI device is shut down), the device is *not* automatically attached back to dom0.
|
|
||||||
This is an intended feature.
|
This is an intended feature.
|
||||||
A device which was previously assigned to a VM less trusted than dom0
|
A device which was previously assigned to a VM less trusted than dom0 (which, in Qubes, is *all* of them) could attack dom0 if it were automatically reassigned there.
|
||||||
(which, in Qubes, is *all* of them) could attack dom0 if it were automatically reassigned there.
|
|
||||||
|
|
||||||
In order to re-enable the device in dom0, either:
|
In order to re-enable the device in dom0, either:
|
||||||
|
|
||||||
|
@ -260,8 +224,7 @@ In order to re-enable the device in dom0, either:
|
||||||
|
|
||||||
or
|
or
|
||||||
|
|
||||||
* Go to the sysfs (`/sys/bus/pci`), find the right device, detach it from the
|
* Go to the sysfs (`/sys/bus/pci`), find the right device, detach it from the pciback driver, and attach it back to the original driver.
|
||||||
pciback driver, and attach it back to the original driver.
|
|
||||||
Replace `<BDF>` with your device, for example `00:1c.2`:
|
Replace `<BDF>` with your device, for example `00:1c.2`:
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
|
|
Loading…
Reference in New Issue
Block a user