Subsections of Commands
proxcli config
Subsections of proxcli config
Create proxcli config file
Options
| option | description | Allowed values |
|---|---|---|
| hosts | coma separated list of proxmox nodes ip addresses / host names | string |
| user | administrator username (only pam at the moment) | string |
| password | password | string |
Examples
- Create a proxmox config file
proxcli config create --hosts "pve1,pve2,pve3" --user root@pam --password rootpasswordShow proxcli configuration
Options
|option|description|Allowed values|
Examples
- show proxcli configuration
proxcli config showVirtual machines
Subsections of Virtual machines
List virtual machines
Options
| option | description | Allowed values |
|---|---|---|
| filter-name | apply a regex filter on virtual machines name | regex string |
| output-format | output the result of the list command in the selected format (default to table) | table or yaml or json |
| proxmox-node | coma separated list of proxmox nodes where we will search for virtual machines | node1,node2,node3 |
| status | virtual machine status | running or stopped |
Examples
- This command will output an unfiltered list of virtual machines available on the cluster
proxcli vms list- This command will output a list of virtual machines on selected nodes
proxcli vms list --proxmox-nodes pve1,pve2- This commands will output a list of virtual machines filterd by a regular expression applied on virtual machine name
proxcli vms list --filter-name "^b4p-"- This command will output a list of virtual machines with the specified status (comma separated list possible)
proxcli vms list --status running- You can combine the different filters together. This command show running vms matching a filter on virtual machine name on a specified node
proxcli vms list --status running --proxmox-node pve2 --filter-name ".*powerdns.*"output formating
Clone a virtual machine template
Options
| option | description | Allowed values |
|---|---|---|
| vmid | virtual machine template id to be cloned | integer |
| name | virtual machine clone name | regex string |
| vm-description | virtual machine clone description | string |
| full-clone | A full clone VM is a complete copy and is fully independant to the original VM or VM Template, but requires the same disk space as the original. | N/A |
| storage | storage name (local or remote) where the virtual machine disk image will be cloned | string |
| target | the target proxmox node name where the cloned virtual machine will be assigned | string |
| block | will the clone process will block until it end ? It make sense on slow remote storage (like on NAS), we wait for the clone task to finish before doing the next clone. This prevent IO saturation on storage | N/A |
| duplicate | how many duplicate do we need ? each duplicate will get his name suffixed by an index. | integer |
vmid and name are mutualy exclusive
Examples
- Clone a virtual machine template
proxcli vms clone --vmid 111 --name test --vm-description test --full-clone --block --target pve2 --storage b4papp - Clone a virtual machine template with cloud enabled template
This is the same command as for a normal clone but you need additional steps to set user, password, ssh public key and network configuration. Try the following commands after cloning finishes.
proxcli vms set --vmid 112 --ciuser myuser --cipassword mypassword --ipconfig "ip=dhcp" --sshkey "$(cat ~/.ssh/id_rsa.pub)"
proxcli vms set --vmid 112 --cores 4 --memory 4096
proxcli vms resize --vmid 112 --disk virtio0 --size "50G"Make sure you already have a cloud init enabled template on your proxmox nodes. You can find in the next section an example of creating an ubuntu server cloud init enabled image.
you will need virt-customize tool on your proxmox nodes in order to install qemu-guest-agent (install it with apt install libguestfs-tools) you also need an ubuntu cloud-init enabled image. You can find one here
virt-customize -a lunar-server-cloudimg-amd64.img --install qemu-guest-agent
virt-customize -a lunar-server-cloudimg-amd64.img --run-command "echo -n > /etc/machine-id"Those commands will help you create an ubuntu cloud init enabled virtual machine template. Adjust the variables at the begining of the script so it match your needs.
vmid=$(pvesh get /cluster/nextid)
isopath="/mnt/pve/isos/template/iso/lunar-server-cloudimg-amd64.img"
templatename="lunar-server-cloudinit"
memory=2048
storage="b4papp"
scsihw="virtio-scsi-pci"
devname="virtio"
disksize="5G"
qm create $vmid --memory ${memory} --name ${templatename} --net0 virtio,bridge=vmbr0
qm set $vmid --agent enabled=1
qm importdisk $vmid $isopath $storage
qm set $vmid --scsihw $scsihw --${devname}0 ${storage}:${vmid}/vm-${vmid}-disk-0.raw
qm set $vmid --ide2 ${storage}:cloudinit
qm set $vmid --boot c --bootdisk ${devname}0
qm resize $vmid ${devname}0 ${disksize}
qm set ${vmid} --serial0 socket --vga serial0
qm template ${vmid}On some shared storages (like Synology NFS share), you may encounter the following error with qm template command:
/usr/bin/chattr: Operation not supported while reading flags on /mnt/pve/b4papp/images/111/base-111-disk-0.rawYou don’t have to do anything, it is just a limitation in synology hardening. But it does not affect the creation of a template.
Set virtual machine parameters
Options
| option | description | Allowed values |
|---|---|---|
| vmid | The (unique) ID of the VM. | integer |
| vmname | Virtual Machine exact name | string |
| cores | The number of cores per socket. | integer |
| sockets | The number of CPU sockets. | integer |
| cpulimit | Limit of CPU usage. NOTE: If the computer has 2 CPUs, it has total of ‘2’ CPU time. Value ‘0’ indicates no CPU limit. | integer |
| memory | Amount of RAM for the VM in MiB. This is the maximum available memory when you use the balloon device. | integer |
| ipconfig | cloud-init: Specify IP addresses and gateways for the corresponding interface. IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified. The special string ‘dhcp’ can be used for IP addresses to use DHCP, in which case no explicit gateway should be provided. For IPv6 the special string ‘auto’ can be used to use stateless autoconfiguration. This requires cloud-init 19.4 or newer. If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using dhcp on IPv4. | string |
| cipassword | cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords. | string |
| ciuser | cloud-init: User name to change ssh keys and password for instead of the image’s configured default user. | string |
| citype | Specifies the cloud-init configuration format. The default depends on the configured operating system type ostype. We use the nocloud format for Linux, and configdrive2 for windows. | string |
| boot | Specify guest boot order. Use the ‘order=’ sub-property as usage with no key or ’legacy=’ is deprecated. | string |
| sshkey | cloud-init: Setup public SSH key | string |
Set parameters is a small subset of what is available in the proxmox API. I (maybe) will had more parameters later.
Examples
assert we have already cloned a cloud init enabled ubuntu virtual machine with the following command:
proxcli vms clone --vmid 100 --full-clone --block --name ubuntu-cloud-init-clone --vm-description exemple_clone --target pve1 --storage b4pappthe cloned vm is not started yet
- cloud init configuration of a vm cloned from a cloud init enabled ubuntu virtual machine
proxcli vms set --vmid 110 --ciuser myuser --cipassword mypassword --ipconfig "ip=dhcp" --sshkey "$(cat ~/.ssh/id_rsa.pub)"- adjust resources of a cloned virtual machine
proxcli vms set --vmid 110 --cores 4 --memory 4096- start the cloned virtual machine
proxcli vms start --vmid 111Resize virtual machine disk
Options
| option | description | Allowed values |
|---|---|---|
| vmid | the virtual machine id from which you want to resize the disk | string |
| vmname | the virtual machine name from which you want to resize the disk | string |
| filter-name | regex selector applied on virtual machines names you want to resize the disk | string |
| disk | the disk name in virtual machine config (usualy the one associated with bootdisk) | string |
| size | The desired size of the disk (Ex: 50G) | string |
vmid, vmname and filter-name are mutualy exclusive
Examples
- resize the boot disk
proxcli vms resize --vmid 111 --disk virtio0 --size "60G"Delete virtual machine
Options
| option | description | Allowed values |
|---|---|---|
| vmid | the virtual machine id you want to delete | string |
| filter-name | a regex applied on virtual machine name | string |
| confirm | does not ask for confirmation | N/A |
| block | block the command until it finished | N/A |
Examples
- delete a virtual machine by its id
proxcli vms delete --vmid 111- delete all virtual machines matching the specified regex
proxcli vms delete --filter-name "^b4p"- delete all virtual machines matching the specified regex without any confirmation and wait for all deletion to be finished before exiting
proxcli vms delete --filter-name "^b4p" --confirm --blockDump virtual machine config
Arguments
| argument | description | Allowed values |
|---|---|---|
| vmid | virtual machine from which we want to dump the config | integer |
Examples
- dump config of a virtual machine
proxcli vms dump_config 111Migrate a virtual machine
Options
| option | description | Allowed values |
|---|---|---|
| vmid | The virtual machine id to migrate | integer |
| filter-name | A regex string to select multiple virtual machines to be migrated | string regex |
| target-node | the proxmox target node name | string |
vmid and filter-name are mutualy exclusive
Examples
You can’t migrate a running virtual machine. The only way to do live migration is through cluster ha resource migrate. See cluster ha section
- migrate a virtual machine from one proxmox node to other
proxcli vms migrate --vmid 111 --target-node pve2 - migrate multiples virtual machines by selecting the with a regex filter on their name
proxcli vms migrate --filter-name "^b4p" --target-node pve2Display the next available vm id
Options
| option | description | Allowed values |
|---|
Examples
- Display the next available virtual machine id
proxcli vms nextIdvirtual machine tags
Subsections of virtual machine tags
List virtual machine tags
Options
| option | description | Allowed values |
|---|
Examples
- list all virtual machine tags
proxcli vms tags listSet virtual machine tags
Options
| option | description | Allowed values |
|---|---|---|
| vm-tags | coma separated list of tags | string |
| filter-name | regex applied to match virtual machine names | string |
| set-mode | whether we can append or replace tags. default to replace | string |
Examples
- Set tags for all vms name matching the specified regex
proxcli vms tags set --vm-tags "template,ubuntu" --filter-name "^ubuntu-cloud"- append tags for all vms name matching the specified regex
proxcli vms tags set --vm-tags "newtag" --filter-name "^ubuntu-cloud" --set-mode appendSet virtual machine status
Description
Virtual machine status commands use the same options interface for every desired status. The following options table apply to all status commands such as start, stop, suspend or reset.
Options
| option | description | Allowed values |
|---|---|---|
| vmid | the virtual machine id you want to delete | string |
| filter-name | a regex applied on virtual machine name | string |
Examples
- Start virtual machine
proxcli vms start --vmid 101- Stop virtual machine
proxcli vms stop --vmid 101- Suspend all virtual machines with name matching specified regex
proxcli vms suspend --filter-name "^b4p"- Reset virtual machine
proxcli vms reset --vmid 101Wait for virtual machine status
Description
wait for virtual machines to reach the desired status (stopped, running …)
Options
| option | description | Allowed values |
|---|---|---|
| vmid | the virtual machine id you want to wait for reaching the desired status | string |
| name | the virtual machine name you want to wait for reaching the desired status | string |
| filter-name | a regex applied on virtual machine name you want to wait for reaching the desired status | string |
| status | the desired status | string |
vmid, name and filter-name are mutualy exclusive
Examples
- wait for every virtual machine with name start with test to be stopped
proxcli vms wait_for_status --filter-name "^test" --status "stopped"Cluster
Subsections of Cluster
cluster status
Options
| option | description | Allowed values |
|---|---|---|
| output-format | format the output to one of table, json or yaml | string |
Examples
- Show cluster status
proxcli cluster statusCluster log
Options
| option | description | Allowed values |
|---|---|---|
| output-format | format the output to one of table, json or yaml | string |
| max-items | max number of log lines we grab from nodes | integer |
| proxmox-nodes | comma separated list of nodes issuing the log lines | string |
| severities | filter logs by severities. this option is a coma separated list of severities labels (panic,alert,critical,error,warning,notice,info,debug) | string |
Examples
- Show cluster log
proxcli cluster log- Show errors searching in the last 1000 logs lines
proxcli cluster log --severities critical,error,warning --max-items 1000High availibility
Definition
High Availability ensures that a VM will stay running even if an individual node is shut down. This means that if the device is either powered off or has any sort of issue, the VM will automatically migrate to another node and start there. This will use all nodes to ensure the VMs configured will stay running as close to 100% of the time as possible.
source: https://www.wundertech.net/how-to-set-up-a-cluster-in-proxmox/
Virtual machine HA in proxmox depend on two concepts:
- groups
- resources
An HA group is a logical configuration specifying on which nodes the vm can be migrated when a proxmox node fail. HA resouces is a virtual machine associated with an HA group. Several resources can be created and associated with a group.
Subsections of High availibility
Cluster ha groups
Subsections of Cluster ha groups
List cluster HA group
Options
| option | description | Allowed values |
|---|
Examples
- list cluster ha groups
proxcli cluster ha groups listCreate an HA cluster group
Options
| option | description | Allowed values |
|---|---|---|
| group | Cluster ha group name to be created | string |
| proxmox-nodes | On which proxmox nodes the group should apply | string |
| restricted | The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior. | N/A |
| nofailback | Resources bound to restricted groups may only run on nodes defined by the group. The resource will be placed in the stopped state if no group node member is online. Resources on unrestricted groups may run on any cluster node if all group members are offline, but they will migrate back as soon as a group member comes online. One can implement a preferred node behavior using an unrestricted group with only one member. | N/A |
Examples
- create an HA group for application powerdns. the group will use proxmox nodes pve1 and pve2 and make sure the group resources will stay on those 2 nodes
proxcli cluster ha groups create --group powerdns --nofailback --proxmox-nodes "pve1,pve2"Delete an HA cluster group
Options
| option | description | Allowed values |
|---|---|---|
| group | Cluster ha group name to be deleted | string |
It is impossible to delete a group with associated resources. You must first delete resources from group and then delete the group
Examples
- delete a cluster HA group
proxcli cluster ha groups delete --group powerdnsCluster HA resources
Subsections of Cluster HA resources
List cluster ha resource
Options
| option | description | Allowed values |
|---|
Examples
- List cluster resources
proxcli cluster ha resources listAdd cluster ha resource
Options
| option | description | Allowed values |
|---|---|---|
| group | the cluster ha group this resource belong to | string |
| vmid | the virtual machine id you want to assign to the resource | string |
| name | the cluster ha resource name | string |
| comment | additional information for this resource | N/A |
| state | Requested resource state. The CRM reads this state and acts accordingly. Please note that enabled is just an alias for started. (disabled | enabled |
| max-relocate | Maximal number of service relocate tries when a service failes to start | integrer |
| max-restart | Maximal number of tries to restart the service on a node after its start failed. | integer |
Examples
- create a cluster ha resource with minimal informations
proxcli cluster ha resources add --group gitlab --vmid 105Delete cluster ha resource
Options
| option | description | Allowed values |
|---|---|---|
| vmid | the virtual machine id you want to remove from resources | string |
| filter-name | A regex applied on virtual machine name used to select matching virtual machines to remove from resources | string |
Examples
- Delete a single cluster ha resource by virtual machine id
proxcli cluster ha resources delete --vmid 105- Delete all cluster ha resources (not recomended)
proxcli cluster ha resources delete --filter-name "^.*$"Migrate cluster ha resource
Options
| option | description | Allowed values |
|---|---|---|
| vmid | The virtual machine id to migrate | integer |
| filter-name | a regex on virtual machines name used to select multiples virtual machines to be migrated | string |
| proxmox-node | the target node to migrate the virtual machines to | string |
| block | wait for each resources to finish migration before starting another one (sequential mode) | string |
vmid and filter-name are mutualy exclusive
Examples
- migrate a cluster ha resource to another node
proxcli cluster ha resources migrate --vmid 125 --proxmox-node pve1- migrate multiple cluster ha resources to another node
proxcli cluster ha resources migrate --filter-name "^b4p-powerdns" --proxmox-node pve1- migrate multiple cluster ha resources to another node waiting for each resource to finish migration
proxcli cluster ha resources migrate --filter-name "^b4p-powerdns" --proxmox-node pve1 --blockRelocate cluster ha resource
Options
| option | description | Allowed values |
|---|---|---|
| vmid | The virtual machine id to be relocated | integer |
| filter-name | a regex on virtual machines name used to select multiples virtual machines to be relocated | string |
| proxmox-node | the target node to relocate the virtual machines to | string |
| block | wait for each resources to finish relocation before starting another one (sequential mode) | string |
vmid and filter-name are mutualy exclusive
Examples
- relocate a cluster ha resource to another node
proxcli cluster ha resources relocate --vmid 125 --proxmox-node pve1- relocate multiple cluster ha resources to another node
proxcli cluster ha resources relocate --filter-name "^b4p-powerdns" --proxmox-node pve1- relocate multiple cluster ha resources to another node waiting for each resource to finish relocation
proxcli cluster ha resources relocate --filter-name "^b4p-powerdns" --proxmox-node pve1 --blockNodes
Subsections of Nodes
List cluster nodes
Options
| option | description | Allowed values |
|---|---|---|
| filter-name | apply a regex filter on nodes names | string |
| output-format | one of table, json or yaml. Default to table | string |
Examples
- List all cluster nodes
proxcli nodes listNodes Networks
Subsections of Nodes Networks
List nodes networks
Options
| option | description | Allowed values |
|---|---|---|
| proxmox-nodes | coma separated list of proxmox nodes from which we want to grab networks | string |
| output-format | format to display networks list (json, yaml or table) | string |
Examples
- Show networks from a selected list of nodes
proxcli nodes nodes networks list --proxmox-nodes "pve1,pve2"Cluster storages
Subsections of Cluster storages
List cluster storages
Options
| option | description | Allowed values |
|---|
Examples
- List cluster storages
proxcli cluster storages listUpload disk image or iso file to cluster storage
Options
| option | description | Allowed values |
|---|---|---|
| file | the path to the local filename to be upload | string |
| storage | the configured storage name in proxmox node | string |
| proxmox-node | proxmox node name | string |
| content | the file content type (iso or images) | string |
Examples
- upload iso file to cluster storage
Nodes storages content
Subsections of Nodes storages content
List nodes storages content
Options
| option | description | Allowed values |
|---|---|---|
| storage | the node storage name (local-lvm, isos, …) | string |
| proxmox-node | proxmox node name | string |
| output-format | output format can be one of table, json, yaml (default to table) | string |
| content-type | filter by content column (iso, tzst, raw, qcow2 ….) | string |
| content-format | filter by content format. Can be one of iso,images,backup,vztmpl | string |
| filter-orphaned | can be YES (orphaned content), NO (no orphaned content), N/A (not applicable to orphaned content). This is a coma separated list. Default to YES,NO,N/A | string |
Examples
- List specific node storage content
proxcli nodes storages content list --node pve1 --storage local-lvm- List storage content filtered by images content-type in raw format
proxcli nodes storages content list --node pve1 --storage b4papp --content-type raw --content-format images List nodes storages content
Options
| option | description | Allowed values |
|---|---|---|
| storage | the node storage name (local-lvm, isos, …) | string |
| proxmox-node | proxmox node name | string |
| output-format | output format can be one of table, json, yaml (default to table) | string |
| content-type | filter by content column (iso, tzst, raw, qcow2 ….) | string |
| content-format | filter by content format. Can be one of iso,images,backup,vztmpl | string |
| filter-orphaned | can be YES (orphaned content), NO (no orphaned content), N/A (not applicable to orphaned content). This is a coma separated list. Default to YES,NO,N/A | string |
Examples
- List specific node storage content
proxcli nodes storages content list --node pve1 --storage local-lvm- List storage content filtered by images content-type in raw format
proxcli nodes storages content list --node pve1 --storage b4papp --content-type raw --content-format images Show cluster nodes tasks
Options
| option | description | Allowed values |
|---|---|---|
| proxmox-nodes | coma separated list of proxmox nodes from which we want to grab tasks | string |
Examples
- Show tasks on a list of proxmox nodes
proxcli nodes tasks --proxmox-nodes "pve1,pve2"proxcli invetory
Subsections of proxcli invetory
Show computed proxcli inventory
Options
| option | description | Allowed values |
|---|---|---|
| filter-name | filter inventory by a regex applyed on virtual machines names | string |
| exclude-tag | exclude tags from inventory. Comma separated list of tags | string |
| output-format | output format is one of (json, yaml) | string |
Examples
- show computed inventory
proxcli inventory show- show computed inventory with only virtual machine names starting by test
proxcli inventory show --filter-name "^test"- show computed inventory with only virtual machine names starting by test and exclude production tag
proxcli inventory show --filter-name "^test" --exclude-tag "production"Save proxcli inventory file
Options
| option | description | Allowed values |
|---|---|---|
| exclude-tags | a list of coma separated tags to be excluded from the ansible inventory file | string |
| filter-name | a regex applied on host names so only matching host names will be saved in the ansible inventory file | string |
| output-format | one of json or yaml | string |
| path | full path of the ansible inventory file | string |
Examples
- save a proxmox inventory file in yaml format
proxcli inventory create --path ./inventory.yaml- save a proxmox inventory file in yaml format with host names matching regex “^test”
proxcli inventory create --path ./inventory.yaml --filter-name "^test"












































