<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://iriarte.it/feed.xml" rel="self" type="application/atom+xml" /><link href="https://iriarte.it/" rel="alternate" type="text/html" /><updated>2026-03-11T15:35:07-03:00</updated><id>https://iriarte.it/feed.xml</id><title type="html">Doing the blog thing</title><subtitle>These are personal notes I believe would like to recall at some point in time, sharing them to the wind...</subtitle><entry><title type="html">Openstack Cloud Images</title><link href="https://iriarte.it/datacenter/2025/11/11/Openstack-Cloud-Images.html" rel="alternate" type="text/html" title="Openstack Cloud Images" /><published>2025-11-11T00:00:00-03:00</published><updated>2025-11-11T00:00:00-03:00</updated><id>https://iriarte.it/datacenter/2025/11/11/Openstack-Cloud-Images</id><content type="html" xml:base="https://iriarte.it/datacenter/2025/11/11/Openstack-Cloud-Images.html"><![CDATA[<p>Once you have your shiny Openstack environment up &amp; running you would like to inmediately start provisioning virtual machines. The starting point are OS images that will provide the base copy of Operating System to use for your new virtual machines.</p>

<p>Images include both the OS disk data and what would normally be virtual machine properties in other virtualization platforms.</p>

<p>I’ll provide here a procedure to prepare production grade images coming from the usual Linux Distribution providers.</p>

<!--more-->

<h1 id="cloud-images">Cloud Images</h1>

<p>Enter <a href="https://docs.openstack.org/image-guide/obtain-images.html">“Cloud Images”</a>. These OS images are basically instances prepared/curated by the different Linux distribution projects (or companies) ready to boot in a cloud environment (AWS, Azure, GCP, Openstack, etc). You can treat them as secure as the ISO installer you download for your favorite distribution project, they have a minimal footprint and usually include cloud-init to automate the deployment process.</p>

<p>In Openstack, cloud-init is critical for deployment as you don’t want to resize your boot disk, add users or setup TCP/IP by hand. Also QEMU Guest Agent is highly recommended when virtualizing on KVM to enable consistent backups &amp; allow for some operational inventory automation.</p>

<p>Paid Linux distributions (like SLES or RHEL) also provide cloud images. I can’t seem to find a Windows Server 2025 Cloud image to download (shame on you Microsoft!).</p>

<h1 id="configuration">Configuration</h1>

<p>Actual virtual machine definition needs to be set as properties during the disk image import process. We’ll be defining them as that is not part of the Cloud Images files you download.</p>

<p>Some attributes I’m proposing here provide proper documentation (name, version, licensing, etc), others provide performance best practices (machine type, NIC/HBA types) or provide operational best practices (backup related parameters).</p>

<h1 id="disk-format">Disk format</h1>

<p>In my case, I’m using Ceph RBD as backend for Cinder. If you tell Cinder/Nova to create a volume from a qcow2 image it is going to first download the image, convert it to raw then re-upload it to Ceph. This consumes a lot more space in Ceph, more network bandwidth and is prone to timeouts if conversion and upload takes too long.</p>

<p>If you’re using a filesystem backend go with QCOW2 but if you’re using Ceph RBD like me, go with RAW in order to benefit from Copy-on-Write cloning of images in both Cinder and Nova &amp; quicker instantiation times.</p>

<h1 id="tooling">Tooling</h1>

<p>Depending on the machine you’re working on, install guestfs-tools. This will allow us to customize the image.</p>

<p>openSUSE:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>zypper <span class="nb">install</span> <span class="nt">-y</span> guestfs-tools</code></pre></figure>

<p>Ubuntu:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt <span class="nb">install</span> <span class="nt">-y</span> libguestfs-tools whois</code></pre></figure>

<h1 id="free-images">Free Images</h1>

<p>Find a spot to download files</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">mkdir </span>cloud-images
<span class="nb">cd </span>cloud-images</code></pre></figure>

<h2 id="opensuse">openSUSE</h2>

<p>My personal favorite, we’ll download Leap which is very similar to SLES (the commercial version of the distribution). Other alternatives are Tumbleweed (bleeding edge) or Slowroll (“I want something newer, but don’t get crazy”).</p>

<p>It already supports cloud-init, we just enable support for the PTP device to be used by chrony as time reference. The KVM host basically provides it for the guests to <a href="https://www.libertysys.com.au/2024/04/vm-timekeeping-using-the-ptp-hardware-clock-on-kvm/">sync time</a> and contrary to what happens with VMware Guest Tools, QEMU Guest Agent doesn’t provide time sincronization capabilities.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>Leap-16.0-Minimal-VM.x86_64-Cloud.qcow2
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-opensuse-leap-16.0-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://download.opensuse.org/repositories/openSUSE:/Leap:/16.0:/Images/images/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

<span class="c"># Fix PTP synchronization</span>
<span class="nb">echo </span>ptp_kvm <span class="o">&gt;</span> /tmp/ptp_kvm.conf
<span class="nb">sudo </span>virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--copy-in</span> /tmp/ptp_kvm.conf:/etc/modules-load.d/

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'opensuse'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'16.0'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'opensuse'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"openSUSE Leap 16.0 - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="rocky-linux">Rocky Linux</h2>

<p>Usual tool for the CentOS refugees. Cloud-init is available out of the box.</p>

<h3 id="lvm-based-image">LVM based image</h3>

<p>In case you want to have all your filesystems under LVM management, the project provides a specific image. Only drawback is that cloud-init filesystem resize won’t work and manual maneuvering will be required (don’t expect hardcore Linux users to have a problem with that).</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>Rocky-9-GenericCloud-LVM.latest.x86_64.qcow2 
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-rocky-9-lvm-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://dl.rockylinux.org/pub/rocky/9/images/x86_64/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'rocky'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'9.6'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'rocky'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Rocky Linux 9.6 with LVM - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h3 id="bare-partitions-image">Bare partitions image</h3>

<p>This image has bare partitions for operating system filesystems. As with any other Linux image, you can still use LVM for data disks (and probably you should, since hot migration between Ceph pools is not supported by Openstack and LVM-in-guest can save the day when data relocation is required)</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-rocky-9-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://dl.rockylinux.org/pub/rocky/9/images/x86_64/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'rocky'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'9.6'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'rocky'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Rocky Linux 9.6 - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="oracle-enterprise-linux-9">Oracle Enterprise Linux 9</h2>

<p>This is an odd one and I’ve never been a fan. Another RHEL clone with unfair advantages regarding the made-up Oracle licensing rules. Similarly to Ubuntu, you can pay for support or not. Repositories are public.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>OL9U5_x86_64-kvm-b259.qcow2
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-oel-9.5-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://yum.oracle.com/templates/OracleLinux/OL9/u5/x86_64/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'oel'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'9.5'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'oracle'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Oracle Enterprise Linux 9.5 - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="ubuntu-2404">Ubuntu 24.04</h2>

<p>Ubiquitous distribution, if you like Debian you’ll feel at home.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>noble-server-cloudimg-amd64.img
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-ubuntu-24.04.3-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://cloud-images.ubuntu.com/noble/current/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

<span class="c"># Install QEMU Guest Agent</span>
<span class="nb">sudo </span>virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'ubuntu'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'24.04.3'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'ubuntu'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Ubuntu 24.04.3 (Noble) - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="ubuntu-2204">Ubuntu 22.04</h2>

<p>Older &amp; still used version of Ubuntu.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>jammy-server-cloudimg-amd64.img
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-ubuntu-22.04.5-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://cloud-images.ubuntu.com/jammy/current/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
<span class="nb">sudo </span>virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'ubuntu'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'22.04.5'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'ubuntu'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Ubuntu 22.04.5 (Jammy) - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="debian-12">Debian 12</h2>

<p>Omnipresent and still relevant.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>debian-12-generic-amd64.qcow2
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-debian-12-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Download the official image</span>
wget https://cdimage.debian.org/images/cloud/bookworm/latest/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
<span class="nb">sudo </span>virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'debian'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'12.12'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>opensource <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'debian'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"Debian 12.12 (Bookworm) - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h1 id="paid-images">Paid Images</h1>

<h2 id="redhat-enterprise-linux-10">RedHat Enterprise Linux 10</h2>

<p>There are some shops that still require this guy. You’ll have to download the file via the paid portal and place it in the working directory.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>rhel-10.0-x86_64-kvm.qcow2
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-rhel-10.0-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'rhel'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'10.0'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>rhel <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'cloud-user'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"RedHat Enterprise Linux 10.0 - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h2 id="redhat-enterprise-linux-9">RedHat Enterprise Linux 9</h2>

<p>Same but older. File needs to be downloaded and copied to the working directory.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Some helper variables</span>
<span class="nv">IMG</span><span class="o">=</span>rhel-9.6-x86_64-kvm.qcow2
<span class="nv">IMGRAW</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">IMG</span><span class="p">%.qcow2</span><span class="k">}</span><span class="s2">.raw"</span>
<span class="nv">NAME</span><span class="o">=</span>ci-rhel-9.6-x86_64-<span class="si">$(</span><span class="nb">date</span> <span class="s1">'+%Y%m%d.%H%M'</span><span class="si">)</span>

<span class="c"># Convert to RAW</span>
qemu-img convert <span class="nt">-f</span> qcow2 <span class="nt">-O</span> raw <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span>

<span class="c"># Import</span>
openstack image create <span class="se">\</span>
<span class="nt">--disk-format</span> raw <span class="nt">--container-format</span> bare <span class="se">\</span>
<span class="nt">--progress</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_type</span><span class="o">=</span><span class="s1">'linux'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_distro</span><span class="o">=</span><span class="s1">'rhel'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_version</span><span class="o">=</span><span class="s1">'9.6'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_license</span><span class="o">=</span>rhel <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_admin_user</span><span class="o">=</span><span class="s1">'cloud-user'</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_qemu_guest_agent</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">os_require_quiesce</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_require_fsfreeze</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_machine_type</span><span class="o">=</span><span class="s2">"q35"</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_firmware_type</span><span class="o">=</span>uefi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_serial_port_count</span><span class="o">=</span>1 <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_vif_multiqueue_enabled</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_virtio_packed_ring</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_scsi_model</span><span class="o">=</span>virtio-scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>scsi <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">hw_video_model</span><span class="o">=</span>virtio <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">has_auto_disk_config</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--property</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"RedHat Enterprise Linux 9.6 - Cloud Image"</span> <span class="se">\</span>
<span class="nt">--public</span> <span class="nt">--file</span> <span class="k">${</span><span class="nv">IMGRAW</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span></code></pre></figure>

<h1 id="how-to-use-the-images">How to use the images?</h1>

<p>So, after all that preparation, we should be able to instantiate a virtual machine &amp; automatically configure it with cloud-init with the procedure underneath.</p>

<p>It will take the image, create a virtual disk with the proper size for the OS system to boot from and instantiate our new shiny VM with the parameters defined in the image. As a quick example of what can be done with cloud-init, we also install and start a web server with a basic landing page to be published.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">##</span>
<span class="c"># Credentials</span>
<span class="c">##</span>

<span class="nb">source </span>novarc

<span class="c">## </span>
<span class="c"># Helper variables</span>
<span class="c">##</span>
<span class="c"># Network that will host the VM</span>
<span class="nv">TESTNETWORK</span><span class="o">=</span><span class="s2">"tnt01-dflt-internal-net"</span>
<span class="c"># Alternative reference by ID</span>
<span class="nv">TESTNETWORKID</span><span class="o">=</span><span class="si">$(</span>openstack network show <span class="k">${</span><span class="nv">TESTNETWORK</span><span class="k">}</span> <span class="nt">-f</span> json| jq .id|tr <span class="nt">-d</span> <span class="s1">'"'</span><span class="si">)</span>
<span class="c"># VM name</span>
<span class="nv">TESTVMNAME</span><span class="o">=</span><span class="s2">"prd-mysuper-vm-001"</span>
<span class="c"># VM flavor to use</span>
<span class="nv">FLAVOR</span><span class="o">=</span>c4a.4c-8G
<span class="c"># Base image to use</span>
<span class="nv">IMAGE</span><span class="o">=</span>ci-opensuse-leap-16.0-x86_64-20251111.2246
<span class="c"># OS Disk size in GB</span>
<span class="nv">OSDISKSIZE</span><span class="o">=</span>60
<span class="c"># Were we put the temporary files</span>
<span class="nv">TMPDIR</span><span class="o">=</span>tmpdir-<span class="k">${</span><span class="nv">RANDOM</span><span class="k">}</span>
<span class="c"># User account to use</span>
<span class="nv">MYUSER</span><span class="o">=</span>cloudadmin
<span class="c"># Our SSH public key for authentication</span>
<span class="nv">SSHKEY</span><span class="o">=</span><span class="s2">"ssh-ed25519 AAAA...Ica5ij ciro@laptop001"</span>

<span class="c">##</span>
<span class="c"># Creation procedure</span>
<span class="c">##</span>

<span class="c"># We create our cloud-init config file</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="k">${</span><span class="nv">TMPDIR</span><span class="k">}</span>

<span class="nb">cat</span> <span class="o">&gt;</span> <span class="k">${</span><span class="nv">TMPDIR</span><span class="k">}</span>/userdata.yaml <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh">
#cloud-config
hostname: </span><span class="k">${</span><span class="nv">TESTVMNAME</span><span class="k">}</span><span class="sh">
users:
  - name: </span><span class="k">${</span><span class="nv">MYUSER</span><span class="k">}</span><span class="sh">
    ssh-authorized-keys:
      - </span><span class="k">${</span><span class="nv">SSHKEY</span><span class="k">}</span><span class="sh">
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
#packages:
#  - vim
#  - curl
#  - apache2
runcmd:
  # Add repository
  - zypper ar -f -c https://download.opensuse.org/distribution/leap/16.0/repo/oss/x86_64/ leap-16-oss
  - zypper in -y vim curl apache2
  # Enable and start Apache
  - systemctl enable apache2
  - systemctl start apache2
  # Create Simple index.html
  - ['sh', '-c', 'echo "&lt;h1&gt;Hello world!&lt;/h1&gt;" &gt; /srv/www/htdocs/index.html']
- # Firewall is not present
  #- firewall-cmd --permanent --add-service=http
  #- firewall-cmd --reload
</span><span class="no">EOF

</span><span class="c"># Create the VM instance</span>
openstack server create <span class="se">\</span>
<span class="nt">--image</span> <span class="k">${</span><span class="nv">IMAGE</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--flavor</span> <span class="k">${</span><span class="nv">FLAVOR</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--network</span> <span class="k">${</span><span class="nv">TESTNETWORK</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--use-config-drive</span> <span class="se">\</span>
<span class="nt">--user-data</span> <span class="k">${</span><span class="nv">TMPDIR</span><span class="k">}</span>/userdata.yaml <span class="se">\</span>
<span class="nt">--boot-from-volume</span> <span class="k">${</span><span class="nv">OSDISKSIZE</span><span class="k">}</span> <span class="se">\</span>
<span class="k">${</span><span class="nv">TESTVMNAME</span><span class="k">}</span>

<span class="c"># List all the VMs</span>
openstack server list</code></pre></figure>

<h1 id="improvement-opportunities">Improvement opportunities</h1>

<p>Well, the shared procedure builds a good starting point. Things I see need to be addressed:</p>

<h2 id="1--ptp_kvm-for-all-the-guests">1- ptp_kvm for all the guests</h2>

<p>The kernel module seems to be supported by RHEL &amp; clones like Rocky or OEL:</p>
<ul>
  <li><a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/chap-kvm_guest_timing_management">KVM Guest Timing Management</a></li>
</ul>

<p>It doesn’t seem to be supported by Ubuntu. You’re left either with NTP to network connected servers or no synchronization (poor choice):</p>
<ul>
  <li><a href="https://bugs.launchpad.net/ubuntu/+source/linux-kvm/+bug/1896311">LP #1896311</a></li>
  <li><a href="https://bugs.launchpad.net/ubuntu/+source/linux-meta-kvm/+bug/1950878">LP #1950878</a></li>
</ul>

<p>Need to find time to test and document the configuration procedure</p>

<h2 id="2--os-licensing">2- OS licensing</h2>

<p>Additional configuration is required to properly match paid guest instances to a subset of nodes in a given Openstack deployment. I have identified two options (Image properties + Host Aggregates or traits), but actual testing is required.</p>

<h2 id="3--windows-server-guests">3- Windows Server guests</h2>

<p>It appears to still be a necessary evil in 2025 for large organizations. Need to find time to create a procedure for image creation, it should include Cloudbase-init as alternative to cloud-init.</p>]]></content><author><name>Ciro Iriarte</name></author><category term="Datacenter" /><category term="Virtualization" /><category term="Openstack" /><summary type="html"><![CDATA[Walkthrough of the process to import Cloud Images to Openstack.]]></summary></entry><entry><title type="html">PVE Cloud Images</title><link href="https://iriarte.it/datacenter/2025/02/13/PVE-Cloud-Images.html" rel="alternate" type="text/html" title="PVE Cloud Images" /><published>2025-02-13T00:00:00-03:00</published><updated>2025-02-13T00:00:00-03:00</updated><id>https://iriarte.it/datacenter/2025/02/13/PVE-Cloud-Images</id><content type="html" xml:base="https://iriarte.it/datacenter/2025/02/13/PVE-Cloud-Images.html"><![CDATA[<h1 id="intro">Intro</h1>
<p>Cloud images are pre-configured operating system (OS) images that can be deployed in a cloud environment. The main Linux distributions offer cloud images alongside the regular ISO installers.</p>

<p>Minimal configuration is executed on the public images and imported as PVE templates. The procedure described underneath, expects you to create all the images (some variables defined in a given step are not necessarily re-defined in the next one).</p>

<p>PVE provides the feature of instantiating a VM from a template. We’ll be creating templates using publicly available Cloud Images.</p>

<!--more-->

<blockquote>
  <p><strong>Note:</strong> For ease of use, this knowhow was incorporated into a script available at <a href="https://github.com/ciroiriarte/pve-scripts/blob/master/pve-import-cloud-images">pve-import-cloud-images</a>.</p>
</blockquote>

<h1 id="pre-requisites">Pre-requisites</h1>

<p>A fully configured PVE cluster must be set, with properly configured bridges &amp; datastore to be used.
In one of the PVE nodes, basic tooling is required to modify the images in some cases, most notable, Ubuntu/Debian images are missing the QEMU agent.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">apt <span class="nb">install</span> <span class="nt">-y</span> libguestfs-tools</code></pre></figure>

<h1 id="template-creation">Template creation</h1>

<h2 id="basic-skeleton">Basic skeleton</h2>

<p>To simplify the rollout of the templates, we create a template VM with the common required configuration for cloud-init to work. It will be used to create the distro specific template.</p>

<p>This will include serial console, cloud-init disk configuration, EFI boot mode (and disk) and Q35 machine type. Also, assumes a Linux OS.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">### We set some helper variables</span>
<span class="c"># Where are we placing the template image?</span>
<span class="nv">DSTORE</span><span class="o">=</span><span class="s2">"p-replica3"</span>
<span class="c"># Using Ceph as backend, we must use RAW disk images</span>
<span class="c">#DSKFORMAT=qcow2</span>
<span class="nv">DSKFORMAT</span><span class="o">=</span>raw
<span class="c"># Bridge to connect the vNIC to</span>
<span class="nv">BRIDGE</span><span class="o">=</span>vmbr1
<span class="c"># VM ID to use as starting point</span>
<span class="nv">VMSKELETONID</span><span class="o">=</span>9000
<span class="c"># VM name</span>
<span class="nv">VMSKELETON</span><span class="o">=</span><span class="s2">"tmpl-ci-skeleton"</span>
<span class="c"># Desired bootdisk size. Cloud images are by default small (3-10GB), assign here a sensible size and don't go overboard, secondary data disks are a best practice to simplify data movement and backup/restore operations. When LVM is available, data disks should be added to a secondary volume group and not to the volume group containing the OS.</span>
<span class="nv">BOOTDISKSIZE</span><span class="o">=</span>60G

<span class="c">### Skeleton creation</span>
qm create <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMSKELETON</span><span class="k">}</span> <span class="nt">--cores</span> 2 <span class="nt">--memory</span> 2048 <span class="se">\</span>
<span class="nt">--net0</span> virtio,firewall<span class="o">=</span>1,bridge<span class="o">=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="se">\</span>
<span class="nt">--bios</span> ovmf <span class="nt">--machine</span> q35 <span class="nt">--agent</span> <span class="nv">enabled</span><span class="o">=</span>1 <span class="nt">--ostype</span> l26 <span class="se">\</span>
<span class="nt">--cpu</span> <span class="nv">cputype</span><span class="o">=</span>max <span class="se">\</span>
<span class="nt">--serial0</span> socket <span class="nt">--vga</span> serial0 <span class="se">\</span>
<span class="nt">--efidisk0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:<span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span>,efitype<span class="o">=</span>4m <span class="nt">--scsi1</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:cloudinit</code></pre></figure>

<h2 id="find-a-work-directory">Find a work directory</h2>

<p>We’ll need a work directory to download and manipulate the disk images. Make sure you have enough room.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">mkdir</span> /tmp/cloud-images
<span class="nb">cd</span> /tmp/cloud-images</code></pre></figure>

<h2 id="opensuse">Opensuse</h2>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># We increment VMID starting from the skeleton machine ID</span>
<span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMSKELETONID+1
<span class="c"># Filename in source</span>
<span class="nv">IMG</span><span class="o">=</span>openSUSE-Leap-15.6.x86_64-NoCloud.qcow2
<span class="c"># Name we'll give to the template</span>
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-opensuse-15.6"</span>

<span class="c"># Download the official image</span>
wget https://download.opensuse.org/repositories/Cloud%3A/Images%3A/Leap_15.6/images/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
qemu-img resize <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">BOOTDISKSIZE</span><span class="k">}</span>

<span class="c"># Fix PTP synchronization</span>
<span class="nb">echo </span>ptp_kvm <span class="o">&gt;</span> /tmp/ptp_kvm.conf
virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--copy-in</span> /tmp/ptp_kvm.conf:/etc/modules-load.d/

<span class="c"># Clone our skeleton machine (full clone)</span>
qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
<span class="c"># Import disk in a cluster-wide datastore</span>
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
<span class="c"># We attach the disk, mimic SSD and enable discard/unmap</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
<span class="c"># Define boot options to use the disk</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
<span class="c"># Set VM as template</span>
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="c"># Cleanup</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h3 id="rocky-linux-9">Rocky Linux 9</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMID+1
<span class="nv">IMG</span><span class="o">=</span>Rocky-9-GenericCloud-LVM-9.5-20241118.0.x86_64.qcow2
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-rocky-9"</span>

wget https://dl.rockylinux.org/pub/rocky/9/images/x86_64/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
qemu-img resize <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">BOOTDISKSIZE</span><span class="k">}</span>

qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h3 id="oracle-linux-9">Oracle Linux 9</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMID+1
<span class="nv">IMG</span><span class="o">=</span>OL9U5_x86_64-kvm-b259.qcow2
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-oel-9"</span>

wget https://yum.oracle.com/templates/OracleLinux/OL9/u5/x86_64/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>

qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h3 id="ubuntu-2404">Ubuntu 24.04</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMID+1
<span class="nv">IMG</span><span class="o">=</span>noble-server-cloudimg-amd64.img
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-ubuntu-24.04"</span>

wget https://cloud-images.ubuntu.com/noble/current/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent
qemu-img resize <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">BOOTDISKSIZE</span><span class="k">}</span>

qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h3 id="ubuntu-2204">Ubuntu 22.04</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMID+1
<span class="nv">IMG</span><span class="o">=</span>jammy-server-cloudimg-amd64.img
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-ubuntu-22.04"</span>

wget https://cloud-images.ubuntu.com/jammy/current/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent
qemu-img resize <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">BOOTDISKSIZE</span><span class="k">}</span>

qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h3 id="debian-12">Debian 12</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">let </span><span class="nv">VMID</span><span class="o">=</span>VMID+1
<span class="nv">IMG</span><span class="o">=</span>debian-12-generic-amd64.qcow2
<span class="nv">VMNAME</span><span class="o">=</span><span class="s2">"tmpl-ci-debian-12"</span>

wget https://cdimage.debian.org/images/cloud/bookworm/latest/<span class="k">${</span><span class="nv">IMG</span><span class="k">}</span>
virt-customize <span class="nt">-a</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="nt">--install</span> qemu-guest-agent
qemu-img resize <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">BOOTDISKSIZE</span><span class="k">}</span>

qm clone <span class="k">${</span><span class="nv">VMSKELETONID</span><span class="k">}</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
qm importdisk <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span> <span class="nt">--format</span> <span class="k">${</span><span class="nv">DSKFORMAT</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsihw</span> virtio-scsi-pci <span class="nt">--scsi0</span> <span class="k">${</span><span class="nv">DSTORE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> c <span class="nt">--bootdisk</span> scsi0
qm template <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span>
<span class="nb">rm</span> <span class="k">${</span><span class="nv">IMG</span><span class="k">}</span></code></pre></figure>

<h1 id="vm-provisioning-from-a-template">VM provisioning from a template</h1>

<p>Once the templates are available, you can use them via Web UI, Terraform or CLI. We’ll demostrate how to deploy a VM using CLI on one of the PVE hosts.</p>

<h2 id="vm-creation">VM creation</h2>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">##</span>
<span class="c"># Variables</span>
<span class="c">##</span>

<span class="c"># Template to use</span>
<span class="nv">TMPL</span><span class="o">=</span>tmpl-ci-opensuse-15.6
<span class="c"># Bridge to use</span>
<span class="nv">NETBR</span><span class="o">=</span>frontend
<span class="c"># VM IP</span>
<span class="nv">NETIP</span><span class="o">=</span><span class="s2">"192.168.0.100/24"</span>
<span class="c"># Network gateway</span>
<span class="nv">NETGW</span><span class="o">=</span><span class="s2">"192.168.0.1"</span>
<span class="c"># DNS setup</span>
<span class="nv">DNS</span><span class="o">=</span>8.8.8.8
<span class="nv">DNSSEARCH</span><span class="o">=</span><span class="s2">"my.domain"</span>
<span class="c"># VM name</span>
<span class="nv">NEWVMNAME</span><span class="o">=</span>test001.mydomain.com
<span class="c"># Credentials</span>
<span class="nv">GUESTUSER</span><span class="o">=</span>cloudadmin
<span class="nv">GUESTPASS</span><span class="o">=</span>Cambiar456
<span class="nv">MYKEY</span><span class="o">=</span><span class="s2">"ssh-rsa … Comment"</span>
<span class="nv">KEYFILE</span><span class="o">=</span>/tmp/ssh-%<span class="o">{</span>RANDOM<span class="o">}</span>
<span class="nb">echo</span> <span class="k">${</span><span class="nv">MYKEY</span><span class="k">}</span> <span class="o">&gt;</span> <span class="k">${</span><span class="nv">KEYFILE</span><span class="k">}</span>

<span class="c"># We get the next available VMID</span>
<span class="nv">NEWVM_ID</span><span class="o">=</span><span class="si">$(</span>pvesh get /cluster/nextid<span class="si">)</span>
<span class="c"># Fetching template id using the template name</span>
<span class="nv">TMPL_ID</span><span class="o">=</span><span class="si">$(</span>pvesh get /cluster/resources <span class="nt">--type</span> vm <span class="nt">--noborder</span>|grep <span class="k">${</span><span class="nv">TMPL</span><span class="k">}</span>| <span class="nb">awk</span> <span class="s1">'{ print $1 }'</span>|cut <span class="nt">-f</span> 2 <span class="nt">-d</span> <span class="s2">"/"</span><span class="si">)</span>

<span class="c">##</span>
<span class="c"># Actual provisioning</span>
<span class="c">##</span>

<span class="c"># We clone template to create the new VM, full clone (by default a template spawns linked clones)</span>
qm clone <span class="k">${</span><span class="nv">TMPL_ID</span><span class="k">}</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">NEWVMNAME</span><span class="k">}</span> <span class="nt">--full</span> 1
<span class="c"># Remove and reassign the network device with the correct bridge</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--delete</span> net0
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">-net0</span> virtio,firewall<span class="o">=</span>1,bridge<span class="o">=</span><span class="k">${</span><span class="nv">NETBR</span><span class="k">}</span>
<span class="c"># Define cloud-init variables</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--ciuser</span> <span class="k">${</span><span class="nv">GUESTUSER</span><span class="k">}</span> <span class="nt">--cipassword</span> <span class="k">${</span><span class="nv">GUESTPASS</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--sshkeys</span> <span class="s2">"</span><span class="k">${</span><span class="nv">KEYFILE</span><span class="k">}</span><span class="s2">"</span> <span class="o">&amp;&amp;</span> <span class="nb">rm</span> <span class="k">${</span><span class="nv">KEYFILE</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--ipconfig0</span> <span class="nv">ip</span><span class="o">=</span><span class="k">${</span><span class="nv">NETIP</span><span class="k">}</span>,gw<span class="o">=</span><span class="k">${</span><span class="nv">NETGW</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--nameserver</span> <span class="k">${</span><span class="nv">DNS</span><span class="k">}</span> <span class="nt">--searchdomain</span> <span class="s2">"</span><span class="k">${</span><span class="nv">DNSSEARCH</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Optionally, disable package upgrades in first boot. With repository access (Internet or local mirror), leave it as is.</span>
<span class="c">#qm set ${NEWVM_ID} --ciupgrade 0</span>
<span class="c"># Start the VM</span>
qm start <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span>
<span class="c"># Optionally, attach this session to the VM console (ctrl+o to disconnect afterwards)</span>
qm terminal <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span></code></pre></figure>

<h2 id="optionals">Optionals</h2>

<p>You can add secondary network interfaces and disks to a machine. Example follows.</p>

<h3 id="network">Network</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Defined bridges we're using for the attach operation</span>
<span class="nv">NET1</span><span class="o">=</span>patito
<span class="nv">NET2</span><span class="o">=</span>cangrejo
<span class="nv">NET3</span><span class="o">=</span>zapallo
<span class="c"># Create interfaces and define attachment</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">-net1</span> virtio,firewall<span class="o">=</span>1,bridge<span class="o">=</span><span class="k">${</span><span class="nv">NET1</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">-net2</span> virtio,firewall<span class="o">=</span>1,bridge<span class="o">=</span><span class="k">${</span><span class="nv">NET2</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">-net3</span> virtio,firewall<span class="o">=</span>1,bridge<span class="o">=</span><span class="k">${</span><span class="nv">NET3</span><span class="k">}</span></code></pre></figure>

<h3 id="storage">Storage</h3>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Primary replicated data.</span>
<span class="nv">DSTORE1</span><span class="o">=</span><span class="s2">"p-replica3"</span>
<span class="c"># Secondary data, erasure coding.</span>
<span class="nv">DSTORE2</span><span class="o">=</span><span class="s2">"ec2-1"</span>

<span class="c"># Create vdisk. Long syntax </span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--scsi1</span> <span class="k">${</span><span class="nv">DSTORE1</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span><span class="nt">-disk-2</span>,size<span class="o">=</span>200G,discard<span class="o">=</span>on,ssd<span class="o">=</span>1
<span class="c"># Create vdisk. Simplified syntax, size assumed in GB</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">NEWVM_ID</span><span class="k">}</span> <span class="nt">--scsi1</span> <span class="k">${</span><span class="nv">DSTORE2</span><span class="k">}</span>:500,discard<span class="o">=</span>on,ssd<span class="o">=</span>1</code></pre></figure>]]></content><author><name>Ciro Iriarte</name></author><category term="Datacenter" /><category term="Virtualization" /><category term="Proxmox" /><summary type="html"><![CDATA[Walkthrough of the process to import Cloud Images to Proxmox VE.]]></summary></entry><entry><title type="html">Setting up JunOS devices for Juniper Support Insights</title><link href="https://iriarte.it/datacenter/networking/2023/12/27/setting-up-junos-devices-for-jsi.html" rel="alternate" type="text/html" title="Setting up JunOS devices for Juniper Support Insights" /><published>2023-12-27T00:00:00-03:00</published><updated>2023-12-27T00:00:00-03:00</updated><id>https://iriarte.it/datacenter/networking/2023/12/27/setting-up-junos-devices-for-jsi</id><content type="html" xml:base="https://iriarte.it/datacenter/networking/2023/12/27/setting-up-junos-devices-for-jsi.html"><![CDATA[<h1 id="intro">Intro</h1>

<p>Going back to the “supported” track, after deploying vLWC you need to prepare the devices being monitored. I can’t seem to find a concise guide to properly setup network elements for JSI, so I’m sharing my notes notes after a long trial &amp; error exercise.</p>

<!--more-->

<p>After reading the <a href="https://www.juniper.net/documentation/us/en/software/jsi/jsi-on-jsp-faqs/topics/concept/jsi-faq-list.html">JSI FAQ</a>, my interpretation is that we’ll need to at each network element:</p>

<ol>
  <li>NETCONF-SSH enabled</li>
  <li>SFTP enabled</li>
  <li>A process user account dedicated to JSI/vLWC with sufficient permissions (principle of least privilege)</li>
</ol>

<p>An overview of how data collection works can be found in this <a href="https://www.juniper.net/content/dam/www/assets/flyers/us/en/juniper-support-insights-security-and-privacy-overview.pdf">flyer</a></p>

<figure>
  <a href="/assets/img/2023-12-27_datacollection_overview.png">
  <img src="/assets/img/2023-12-27_datacollection_overview.png" alt="Flow Overview" />
  </a>
  <figcaption><i>Image 1 - Data collection overview</i></figcaption>
</figure>

<h1 id="pre-requirements">Pre-requirements</h1>

<h2 id="ssh-key-pair">SSH key pair</h2>

<p>We need a pair of public/private SSH keys for the user. You can create it on any Linux machine.</p>

<p><strong>Note:</strong> Down the road I found that JunOS won’t accept DSA keys but RSA only.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">ciro.iriarte@jhost01:~&gt; ssh-keygen <span class="nt">-t</span> rsa <span class="nt">-C</span> <span class="s2">"JSI process user"</span> <span class="nt">-f</span> ~/.ssh/id_rsa-jsi
Generating public/private rsa key pair.
Enter passphrase <span class="o">(</span>empty <span class="k">for </span>no passphrase<span class="o">)</span>: &lt;<span class="nt">--</span> make sure you input a passphrase
Enter same passphrase again: 
Your identification has been saved <span class="k">in</span> /home/ciro.iriarte/.ssh/id_rsa-jsi
Your public key has been saved <span class="k">in</span> /home/ciro.iriarte/.ssh/id_rsa-jsi.pub
The key fingerprint is:
SHA256:vMRG09Sd7BR1Q5P7B+9RakR2thSR3Md3bBOtYmm+VVc JSI process user
The key<span class="s1">'s randomart image is:
+---[RSA 3072]----+
|          .. o+&amp;@|
|         o  . B*E|
|        o .  *.=O|
|       + .  = =o+|
|        S  + o =+|
|       o .  . +.+|
|        .    + .o|
|            .   .|
|                 |
+----[SHA256]-----+</span></code></pre></figure>

<p>You’ll use:</p>
<ul>
  <li>the contents of <em>~/.ssh/id_rsa-jsi.pub</em> to setup access in the switches</li>
  <li>the contents of <em>~/.ssh/id_rsa-jsi</em> + <em>the passphrase will</em> be used in the JSI portal to define the access credentials.</li>
</ul>

<h2 id="actual-commands-that-should-be-permitted">Actual commands that should be permitted.</h2>

<p>From the FAQ mentioned before we get that the commands the appliance will execute are:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file list detail
show bgp summary
show chassis alarms
show chassis fpc
show chassis hardware extensive
show chassis routing-engine
show interfaces descriptions
show interfaces terse
show isis adjacency
show ospf neighbor
show rsvp neighbor
show system buffers
show system commit
show system core-dumps
show system license
show system uptime
show version
show vrrp detail
request support information
sftp
</code></pre></div></div>

<h1 id="apstra-managed-switches">Apstra managed switches</h1>

<h2 id="netconf">NETCONF</h2>
<p>NETCONF is already an Apstra requirement, so you should have that already covered. To quickly validate, you can verify the default port is listening for connection requests:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">ciro.iriarte@jhost01:~&gt; nmap <span class="nt">-p</span> 830 ipf01-lf01.&lt;TLD&gt;
Starting Nmap 7.92 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2023-12-27 14:12 EST
Nmap scan report <span class="k">for </span>ipf01-lf01.&lt;TLD&gt; <span class="o">(</span>x.y.z.w<span class="o">)</span>
Host is up <span class="o">(</span>0.0012s latency<span class="o">)</span><span class="nb">.</span>

PORT    STATE SERVICE
830/tcp open  netconf-ssh

Nmap <span class="k">done</span>: 1 IP address <span class="o">(</span>1 host up<span class="o">)</span> scanned <span class="k">in </span>6.58 seconds</code></pre></figure>

<h2 id="sftp">SFTP</h2>
<p>Not sure if SFTP is a requirement for Apstra, but found out that all the switches in the Apstra environment had the SFTP subsystem already enabled. If that’s not your case, a configlet for enabling it is all you need.</p>

<h2 id="process-user">Process user</h2>

<p>In this environment we don’t have TACACS+ (it’s 2023 Juniper team, can we please get LDAP already?). For practical purposes, we’ll use Apstra configlets to create a process users from Apstra.</p>

<h3 id="configlet-creation">Configlet creation</h3>

<ol>
  <li>Reach the configlet creation page.</li>
</ol>
<figure>
  <a href="/assets/img/2023-12-27_configlet00.png">
  <img src="/assets/img/2023-12-27_configlet00.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 2 - Design --&gt; Configlets</i></figcaption>
</figure>
<figure>
  <a href="/assets/img/2023-12-27_configlet01.png">
  <img src="/assets/img/2023-12-27_configlet01.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 3 - Create configlet button "</i></figcaption>
</figure>

<ol start="2">
  <li>Fill in the basic configlet data</li>
</ol>

<figure>
  <a href="/assets/img/2023-12-27_configlet02.png">
  <img src="/assets/img/2023-12-27_configlet02.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 4 - Actual configlet create</i></figcaption>
</figure>

<ol start="3">
  <li>Paste the following content in the body, and click “Create”
    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>system {
 login {
     class jsi-class {
         permissions view-configuration;
         allow-commands "(file list detail)|(show bgp summary)|(show chassis alarms)|(show chassis fpc)|(show chassis hardware extensive)|(show chassis routing-engine)|(show interfaces descriptions)|(show interfaces terse)|(show isis adjacency)|(show ospf neighbor)|(show rsvp neighbor)|(show system buffers)|(show system commit)|(show system core-dumps)|(show system license)|(show system uptime)|(show version)|(show vrrp detail)|(request support information)|(sftp)";
         deny-commands "(clear)|(file)|(file show)|(help)|(load)|(monitor)|(op)|(request)|(save)|(set)|(start)|(test)";
         deny-configuration all;
     }
     user svcjsi {
         full-name "JSI process user";
         uid 2007;
         class jsi-class;
         authentication {
             ssh-rsa "ssh-rsa your-key-super-long-string JSI process user";
         }
     }
 }
}
</code></pre></div>    </div>
  </li>
</ol>

<p><strong>Note:</strong> Make sure UID is unique in your environment. I saw random posts/notes/documentation referencing UID 2000 (for other unrelated implementations), I was super clever changing that to 2001 so I would avoid clashing with config everybody else could be copying/pasting, it turns out whoever implemented Apstra had the same thought.</p>

<h3 id="importing-configlet-to-blueprint">Importing Configlet to blueprint</h3>

<p>To actually apply the previously created configlet to your network elements, you must:</p>

<ol>
  <li>Import configlet to the blueprint, assigning it to the devices of interest. In my case, that would be all the spines and leaf switches.</li>
</ol>

<figure>
  <a href="/assets/img/2023-12-27_import-configlet00.png">
  <img src="/assets/img/2023-12-27_import-configlet00.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 5 - Reach the import section</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-12-27_import-configlet01.png">
  <img src="/assets/img/2023-12-27_import-configlet01.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 6 - Select configlet and assign to devices of interest</i></figcaption>
</figure>

<ol start="2">
  <li>Commit the configuration to be pushed to the devices.</li>
</ol>

<figure>
  <a href="/assets/img/2023-12-27_commit00.png">
  <img src="/assets/img/2023-12-27_commit00.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 7 - Request to push changes to the NEs</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-12-27_commit01.png">
  <img src="/assets/img/2023-12-27_commit01.png" alt="Apstra screenshot" />
  </a>
  <figcaption><i>Image 8 - Fill-in the changelog</i></figcaption>
</figure>

<h1 id="standalone-switches">Standalone switches</h1>

<p>These switches aren’t managed by Apstra or any other manager, also there’s no TACACS+ or similar authentication  or RBAC services (did I mention I would love to have LDAP available to reuse already existing authentication services?). Given they are only 2, manual &amp; local setup should not be a problem.</p>

<p>Connect to the switch with an admin user, in edit mode.</p>

<h2 id="netconf-1">NETCONF</h2>

<p>We enable the service. Cool automation kids recommend the last 3 commands for interoperability, but given vLWC is a closed tool by Juniper for JunOS it might expect the less standard non compliant default configuration. Something to monitor and validate after adding switches to JSI.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set system services netconf ssh
set system services netconf rfc-compliant
set system services netconf yang-compliant 
set system services netconf hello-message yang-module-capabilities advertise-standard-yang-modules
</code></pre></div></div>

<h2 id="sftp-1">SFTP</h2>

<p>SFTP subssystem needs to be enabled, seems to be disabled by default.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set system services ssh sftp-server
</code></pre></div></div>

<h2 id="process-user-1">Process user</h2>

<ol>
  <li>We create the user class with the required access level
    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set system login class jsi-class permissions view-configuration
set system login class jsi-class allow-commands "(file list detail)|(show bgp summary)|(show chassis alarms)|(show chassis fpc)|(show chassis hardware extensive)|(show chassis routing-engine)|(show interfaces descriptions)|(show interfaces terse)|(show isis adjacency)|(show ospf neighbor)|(show rsvp neighbor)|(show system buffers)|(show system commit)|(show system core-dumps)|(show system license)|(show system uptime)|(show version)|(show vrrp detail)|(request support information)|(sftp)"
set system login class jsi-class deny-commands "(clear)|(file)|(file show)|(help)|(load)|(monitor)|(op)|(request)|(save)|(set)|(start)|(test)"
set system login class jsi-class deny-configuration all
</code></pre></div>    </div>
  </li>
  <li>We create the user, using the previously created class. Use your public key not mine!
    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set system login user svcjsi class jsi-class
set system login user svcjsi full-name "JSI process user"
set system login user svcjsi uid 2007
set system login user svcjsi authentication ssh-rsa "ssh-rsa your-key-super-long-string JSI process user"
</code></pre></div>    </div>
  </li>
</ol>

<h2 id="final-commit">Final commit</h2>

<ol>
  <li>Commit the configuration (both service &amp; user)
    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>commit
</code></pre></div>    </div>
  </li>
</ol>

<h1 id="testing-netconf-by-hand">Testing NETCONF by hand</h1>

<p>Before going through the whole JSI apparatus, my recommendation would be to run a basic local test with the user we have created.</p>

<p>Login to a Linux machine with access to the switches and the private key for the user we have created, and try to connect to the switch using NETCONF</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># We start the ssh key agent</span>
<span class="nb">eval</span> <span class="si">$(</span>ssh-agent<span class="si">)</span>

<span class="c"># We add the key to the agent, the command will prompt for the key passphrase</span>
ssh-add ~/.ssh/id_rsa-jsi

<span class="c"># We try a connection on default port</span>
<span class="nv">MYSWITCH</span><span class="o">=</span>host.tld
ssh svcjsi@<span class="nv">$MYSWITCH</span> <span class="nt">-p</span> 830 <span class="nt">-s</span> netconf</code></pre></figure>

<p>Answer should be similar to</p>

<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="c">&lt;!-- No zombies were killed during the creation of this user interface --&gt;</span>
<span class="c">&lt;!-- user svcjsi, class j-jsi-class --&gt;</span>
<span class="nt">&lt;nc:hello</span> <span class="na">xmlns:nc=</span><span class="s">"urn:ietf:params:xml:ns:netconf:base:1.0"</span><span class="nt">&gt;</span>
   <span class="nt">&lt;nc:capabilities&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:netconf:base:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:netconf:capability:candidate:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:netconf:capability:confirmed-commit:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:netconf:capability:validate:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf<span class="ni">&amp;amp;</span>revision=2011-06-01<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:netconf:capability:validate:1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types<span class="ni">&amp;amp;</span>revision=2013-07-15<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>http://xml.juniper.net/netconf/junos/1.0<span class="nt">&lt;/nc:capability&gt;</span>
    <span class="nt">&lt;nc:capability&gt;</span>http://xml.juniper.net/dmi/system/1.0<span class="nt">&lt;/nc:capability&gt;</span>
  <span class="nt">&lt;/nc:capabilities&gt;</span>
  <span class="nt">&lt;nc:session-id&gt;</span>65596<span class="nt">&lt;/nc:session-id&gt;</span>
<span class="nt">&lt;/nc:hello&gt;</span>
]]&gt;]]&gt;</code></pre></figure>

<h1 id="is-that-it">Is that it?</h1>

<p>Well, it should be all for the NE configuration. Now you should be able to login to the JSI portal an onboard devices (Juniper guys, please add support for FQDN in the form besides IP addresses).</p>

<p>Have fun!</p>

<h1 id="more-reference-documents">More reference documents</h1>

<p>In case you’re still reading, these are some documents &amp; notes I’ve reviewed in the process.</p>

<ul>
  <li><a href="https://www.juniper.net/documentation/us/en/software/junos/netconf/topics/topic-map/netconf-ssh-connection.html">Establish an SSH Connection for a NETCONF Session</a></li>
  <li><a href="https://www.tinfoilcipher.co.uk/2021/07/29/ansible-and-juniper-junos-managing-devices-with-ssh-and-netconf/">Managing Juniped devices with SSH and NETCONF</a></li>
  <li><a href="https://www.juniper.net/documentation/en_US/junos/topics/example/access-privileges-individual-operational-mode-commands-configuring.html">Example: Configuring User Permissions with Access Privileges for Operational Mode Commands</a></li>
  <li><a href="https://www.juniper.net/documentation/us/en/software/junos/netconf/topics/task/netconf-session-connecting-to-server.html">Connect to the NETCONF Server Using SSH</a></li>
  <li><a href="https://supportportal.juniper.net/s/article/Junos-Generating-SSH-RSA-keys-locally-on-devices-running-Junos-OS">Technical note on using SSH keys with JunOS</a></li>
</ul>

<h1 id="known-errors">Known errors</h1>

<h2 id="jsi-reporting-wrong-credentials">JSI reporting wrong credentials</h2>

<p>During my tests, I found that JSI web portal didn’t handle credential updates. You could save the changes, you can see the new information in the web portal but it seems it doesn’t reach their backend. I had to delete/recreate the credentials in the web portal for them to work.</p>

<h2 id="netconf-reported-as-not-configured">NETCONF reported as not configured</h2>

<p>During my tests, I could confirm manually that NETCONF was properly setup, but JSI kept reporting:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The NETCONF subsystem is not enabled on the devices with the listed IP addresses. Please check and correct these entries
</code></pre></div></div>

<p>The JSI team reported there was a timer that didn’t allow the NETCONF dialogue to finish properly. This was fixed on their end expanding the timeout configuration, not sure if this is a platform wide parameter or you should request this for your specific nodes/sites/accounts.</p>

<p>You can reference Juniper SR # 2023-1228-038580.</p>]]></content><author><name>Ciro Iriarte</name></author><category term="Datacenter" /><category term="Networking" /><category term="Juniper" /><category term="QFX" /><category term="EX" /><category term="JunOS" /><summary type="html"><![CDATA[Walkthrough NE setup to allow vLWC to connect and extract information.]]></summary></entry><entry><title type="html">Deploying Juniper vLWC on PVE</title><link href="https://iriarte.it/datacenter/networking/2023/12/25/deploying-juniper-vlwc-on-pve.html" rel="alternate" type="text/html" title="Deploying Juniper vLWC on PVE" /><published>2023-12-25T00:00:00-03:00</published><updated>2023-12-25T00:00:00-03:00</updated><id>https://iriarte.it/datacenter/networking/2023/12/25/deploying-juniper-vlwc-on-pve</id><content type="html" xml:base="https://iriarte.it/datacenter/networking/2023/12/25/deploying-juniper-vlwc-on-pve.html"><![CDATA[<h1 id="intro">Intro</h1>

<p>As with any critical infrastructure, technology vendors provide a “call-home” feature for their products. For more than a decade (at least to my knowledge), storage vendors like Hitachi or EMC would provide a solution that would allow the vendor to connect to systems you own for support/assistance departments to directly work on them instead of telling you on the phone what to do. Other expected functionality includes opening a proactive ticket on hardware failure, extract/upload logs or inventory installed base.</p>

<p>This call-home functionality is usually delivered as a piece of software running on a dedicated/physical appliance, as a software to be installed on a general purpose Operating System instance (Windows Server, SLES/RHEL/OEL/Debian) or, these days, a pre-packaged virtual appliance.</p>

<p>Implementing such functionality, is always a good thing to do as it allows the vendor support team to provide quicker response when there are failures.</p>

<!--more-->

<h1 id="the-need">The need</h1>

<p>We were approached by the Juniper Support team, pitching for a WLC appliance which would provide call-home capabilities, and oversee our installed base. This appliance is part of their Juniper Support Insights (JSI) initiative.</p>

<p>One of such physical appliances is en route to one of our sites. Meanwhile, I’m overseeing a lab deployment that could benefit from the quick support and inventory capabilities. Further reviewing this topic, the Juniper team mentioned there’s a virtual version of the appliance that was recently released.</p>

<p>The formal <a href="https://www.juniper.net/documentation/us/en/software/jsi/vlwc-deploy/topics/topic-map/install-verify-vlwc.html">documentation</a> for vSphere environments is pretty straightforward and you can just follow that if you’re working on a supported environment. In my case, the support sidecart environment for the lab doesn’t sport VMware vSphere, but Proxmox VE.</p>

<p>I’ll deploy vLWC on Proxmox VE to oversee QFX switches managed by Apstra &amp; EX switches which comprise the OOBM network for the QFX switches too.</p>

<h1 id="here-be-dragons">Here be dragons.</h1>

<figure>
  <a href="https://education.nationalgeographic.org/resource/here-be-dragons/">
  <img src="/assets/img/2023-12-25_1570-theatrum-orbis-terrarum-map-monsters.jpg" alt="1570 Theatrum Orbis Terrarum Map Monsters" />
  </a>
  <figcaption><i>Image 1 - 1570 Theatrum Orbis Terrarum Map Monsters</i></figcaption>
</figure>

<p>As a word of warning, I should highlight the setup I’m about to execute and share with you is not supported by Juniper and your Service Manager will either frown on you, yell at you or just plainly ignore you (depending on the kind of relationship you have) if you ever request help deploying vLWC on KVM. In our case, I welcome Alejandro for the administrative support!.</p>

<p>Also remember, “not supported” doesn’t mean it’s not technically possible. It just means you’re on your own. I hope Proxmox and/or KVM support is added at some point in the future.</p>

<p>Yes, I’ll get support via a “not supported” appliance. Does it qualify as oximoron?</p>

<h1 id="network-architecture">Network architecture</h1>
<h2 id="lab-sidecart-environment">Lab sidecart environment</h2>
<p>In the lab, we have an environment with the usual set of vendors &amp; technology we work with. That setup includes a EVPN/VXLAN IP Fabric network with several clients being served. That network and the client platforms are expected to be reinstalled or rearranged at any point in time.</p>

<p>To provide some stable essential services, we’ve built a small parallel environment with some switches and PVE servers to run elements that must be available at all times:</p>

<ul>
  <li>Remote access platform</li>
  <li>Authentication Services</li>
  <li>Security related tools</li>
  <li>Monitoring tools</li>
  <li>Firmware &amp; Hardware related solutions</li>
</ul>

<p>With this “sidecart” environment, no matter what is needed to be rearranged in the main environment, we’ll never lose administration capabilities.</p>

<p>For the scope of our discussion, the logical environment looks something like this:</p>

<figure>
  <a href="/assets/img/2023-12-25_lab-environment.svg">
  <img src="/assets/img/2023-12-25_lab-environment.svg" alt="Lab networks overview" />
  </a>
  <figcaption><i>Image 2 - Related management network environment</i></figcaption>
</figure>

<p>Physically, the sidecart is just a pair of Cisco Nexus switches in stack to provide redudant L2 1GbE/10GbE services, and the OOBM network is a pair of Juniper EX switches providing 1GbE connectivity to switches and IPMI server boards.</p>

<p>Logically, the setup is quite involved and provides things like access control and network segmentation via OPNSense providing L3 &amp; Filtering capabilities. We need users to authenticate via username/password + OTP, then we hit the Jump hosts using Guacamole:</p>

<figure>
  <a href="/assets/img/2023-12-25_lab-environment-flows.svg">
  <img src="/assets/img/2023-12-25_lab-environment-flows.svg" alt="Remote access" />
  </a>
  <figcaption><i>Image 3 - Remote access flow</i></figcaption>
</figure>

<p>My natural expectation was to deploy the call-home appliance next to other supporting appliances like Apstra or LibreNMS.</p>

<figure>
  <a href="/assets/img/2023-12-25_lab-environment-plus-vlwc.svg">
  <img src="/assets/img/2023-12-25_lab-environment-plus-vlwc.svg" alt="vLWC in the picture" />
  </a>
  <figcaption><i>Image 4 - Adding vLWC to the picture</i></figcaption>
</figure>

<p>This is relevant for the next section.</p>

<h2 id="required-network-topology">Required network topology</h2>

<p>Reviewing the <a href="https://www.juniper.net/documentation/us/en/software/jsi/vlwc-deploy/topics/concept/system-requirements.html">deployment guide</a>, you can see it mandates 3 networks interfaces:</p>

<table>
  <thead>
    <tr>
      <th>Interface</th>
      <th>Supported IP Address</th>
      <th>Role</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Internal</td>
      <td>IPv4 or IPv6 address</td>
      <td>Used for connection to the switches to be monitored. Default route is also here.</td>
    </tr>
    <tr>
      <td>External</td>
      <td>IPv4 address only</td>
      <td>Used to connect to Death Star (actual call-home)</td>
    </tr>
    <tr>
      <td>Management</td>
      <td>IPv4 address only</td>
      <td>Used for captive portal access (web portal presented by appliance showing status)</td>
    </tr>
  </tbody>
</table>

<p>We could try to comply, and shoe-horn the appliance requirements to our environment:</p>

<figure>
  <a href="/assets/img/2023-12-25_lab-environment-plus-vlwc-forced-compliance.svg">
  <img src="/assets/img/2023-12-25_lab-environment-plus-vlwc-forced-compliance.svg" alt="vLWC in the picture" />
  </a>
  <figcaption><i>Image 5 - Forced compliance</i></figcaption>
</figure>

<p>It doesn’t make me confortable, VMs connected to different segments that should only interact through the firewall was not an option in our case.</p>

<h2 id="appliance-analysis-trial--error">Appliance analysis: Trial &amp; Error</h2>

<p>I did several runs of the deployment on KVM in a vacuum to validate what’s going on with the network setup and analyze what’s required to deploy it with a single interface.</p>

<p>I’ll spare you on the details, but the summary of the findings is:</p>

<h3 id="first-run-3-networks-with-dhcp-internal--external--management"><strong>First run:</strong> 3 networks with DHCP (internal + external + management)</h3>
<p><em>“int”</em> keeps the default route<br />
<em>“ext”</em> has static routes to reach Death Star (JNI)<br />
<em>“cap”</em> is used for captive portal (appliance status page)</p>

<p>The mentioned static routes (in case you’re curious, those segments are owned by AWS &amp; DigitalOcean):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>35.71.174.221 via &lt;ext segment gateway&gt; dev ext 
35.83.44.146 via &lt;ext segment gateway&gt; dev ext 
52.223.32.79 via &lt;ext segment gateway&gt; dev ext 
54.149.98.110 via &lt;ext segment gateway&gt; dev ext 
54.149.222.60 via &lt;ext segment gateway&gt; dev ext 
68.183.248.21 via &lt;ext segment gateway&gt; dev ext 
138.197.235.241 via &lt;ext segment gateway&gt; dev ext 
</code></pre></div></div>
<p><strong>First takeaway:</strong> We might need to massage the appliance to avoid installing the static routes, unless the static routes installation silently fails due to missing EXT interface.</p>

<p>The SSHD &amp; Captive Portal services binding is not limited to any interface in particular.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@ggc-lnx:~# lsof <span class="nt">-n</span> <span class="nt">-sTCP</span>:LISTEN <span class="nt">-i</span> :443
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 20982 root    4u  IPv4 106963      0t0  TCP <span class="k">*</span>:https <span class="o">(</span>LISTEN<span class="o">)</span>
docker-pr 20994 root    4u  IPv6 107636      0t0  TCP <span class="k">*</span>:https <span class="o">(</span>LISTEN<span class="o">)</span>
root@ggc-lnx:~# lsof <span class="nt">-n</span> <span class="nt">-sTCP</span>:LISTEN <span class="nt">-i</span> :22 
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    543 root    3u  IPv4  17619      0t0  TCP <span class="k">*</span>:ssh <span class="o">(</span>LISTEN<span class="o">)</span>
sshd    543 root    4u  IPv6  17621      0t0  TCP <span class="k">*</span>:ssh <span class="o">(</span>LISTEN<span class="o">)</span></code></pre></figure>

<p><strong>Second Takeaway:</strong> we don’t need to fiddle with service binding configuration, all services listen on all interfaces.</p>

<p>So the “don’t answer on EXT” behaviour should most probably come from firewall rules, except it doesn’t (no firewall rule exists to limit ingress SSH requests to a given inteface), the behaviour was a construction of my imagination and there’s not mention of it in the docs.</p>

<p>My ssh test was conducted from a client in the same subnet hosting the EXT interface. My wild guess is that any other remote client connection attempts coming from Internet should fail due to default gateway being installed on INT.</p>

<p>Anyhow kids, don’t place the appliance directly connected to Internet without incoming filters if you can avoid it.</p>

<p><strong>Third Takeaway:</strong> we don’t need to mess with firewall configuration.</p>

<p><strong>Fourth takeaway:</strong> The following screenshot depicts how a healthy appliance looks like.</p>

<figure>
  <a href="/assets/img/2023-12-25_status-deployment-test-3if.png">
  <img src="/assets/img/2023-12-25_status-deployment-test-3if.png" alt="Appliance status" />
  </a>
  <figcaption><i>Image 6 - Appliance status with supported network layout</i></figcaption>
</figure>

<h3 id="second-run-1-network-setup-with-dhcp-internal"><strong>Second run:</strong> 1 network setup with DHCP (internal)</h3>
<p><em>“int”</em> keeps the default route, provides access to captive portal (appliance status page)<br />
<em>“ext”</em> status is reported as failed</p>

<p><strong>Fifth takeway:</strong> Even though the status page show a “Connection Status” errored for “External Network”, you can see that we still have “Juniper Cloud Connected” &amp; “Cloud Provisioned”. My interpretation is that we have an error because there’s no EXT interface that can be setup, but it’s cosmetic since we have Internet access using INT with the default route.</p>

<figure>
  <a href="/assets/img/2023-12-25_status-deployment-test-1if.png">
  <img src="/assets/img/2023-12-25_status-deployment-test-1if.png" alt="Appliance status" />
  </a>
  <figcaption><i>Image 6 - Appliance status with just one interface</i></figcaption>
</figure>

<p>Also I can confirm that after checking the appliance at the OS level, there are no static routes. No adjustments are needed for a single interface deployment to connect to JSI.</p>

<h1 id="the-appliance">The appliance</h1>
<h2 id="first-administrative-steps">First administrative steps.</h2>

<p>You have to request two things to your Juniper support team:</p>
<ul>
  <li>Access to Juniper Support Insights to request the appliance</li>
  <li>Proper permissions to be setup to manage/use the appliance. We have a myriad of support accounts and something was not setup properly for my account in JSI.</li>
</ul>

<h2 id="how-to-request-the-appliance">How to request the appliance.</h2>

<ol>
  <li>Once you have the correct permissions set for your user, head to <a href="https://supportportal.juniper.net/">Juniper Support Portal</a></li>
  <li>Request the appliance via the web portal.</li>
</ol>
<figure>
  <a href="/assets/img/2023-12-25_request_appliance_00.png">
  <img src="/assets/img/2023-12-25_request_appliance_00.png" alt="Collectors section" />
  </a>
  <figcaption><i>Image 7 - Access collectors section</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-12-25_request_appliance_01.png">
  <img src="/assets/img/2023-12-25_request_appliance_01.png" alt="Request" />
  </a>
  <figcaption><i>Image 8 - Click "Request Virtual Lightweight Collector"</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-12-25_request_appliance_02.png">
  <img src="/assets/img/2023-12-25_request_appliance_02.png" alt="Request" />
  </a>
  <figcaption><i>Image 9 - Click "Fill-in and submit form"</i></figcaption>
</figure>

<ol start="3">
  <li>In less than an hour, you should receive an email from <em>do_not_reply@juniper.net</em> with a link to download the appliance. The file is purged in 7 days, after which you would need to re-request the appliance.</li>
</ol>

<h1 id="deployment">Deployment</h1>

<h2 id="what-are-you-receiving-from-juniper">What are you receiving from Juniper?</h2>

<p>When you download the appliance from the Juniper Support site, you will get a OVA file with a name similar to vLWC-${VERSION}-$(SERIAL_NUMBER).ova. In my case it was version 2.3.26.</p>

<p>An OVA file is basically a ZIP file with a VMDK disk image and a OVF file with the definition of the virtual machine to be created, including metadata and definition of what the user should input during rollout (we’ll skip de manifest file for the sake of this discussion).</p>

<p>The VMDK is apparently generic for all the customers, the OVF file includes the serial number specific to your deployment and needs to be incorporated during the instantiation/rollout.</p>

<h2 id="initial-setup">Initial setup</h2>

<p>With OVF customization available via vSphere + VMware Tools some parameters are passed to the VM at instantiation time, but that’s not supported by Proxmox VE. Proxmox VE + cloud-init could provide similar capabilities, but that’s not currently support by the vLWC appliance.</p>

<figure>
  <a href="/assets/img/2023-12-25_missmatch.jpg">
  <img src="/assets/img/2023-12-25_missmatch.jpg" alt="Missmatch" />
  </a>
  <figcaption><i>Image 10 - Autoconfig capabilities, Proxmox VE to the left, Juniper vLWC to the right "</i></figcaption>
</figure>

<p>We would need to modify the original disk before importing it to the virtual machine. I’ll try to make as few modifications as possible.</p>

<ul>
  <li>Required changes
    <ol>
      <li>Network setup comes pre-configured in the appliance for DHCP, given we have DHCP for the sidecart environment, we’ll keep it as is. Other environments might need static addressing, out of scope for the time being.</li>
      <li>In any scenario, a mandatory change for the appliance would be to introduce the serial number for it to properly report to JSI. It requires two entries, a plain text entry which we’ll recover from the downloaded filename and an encrypted entry which we should recover from the OVF file.</li>
    </ol>
  </li>
  <li>Nice to have changes
    <ol>
      <li>We’ll adjust hostname for the sake of proper inventory. I’ve seen the logs of the appliance reporting home the hostname as product name &amp; product version (Juniper guys, that’s not what a hostname is for), currently it doesn’t seem to affect functionality but will monitor and rollback if needed.</li>
      <li>Timezone. The appliance comes preset with UTC timezone, I’ll be adjusting that for our timezone. Will have to monitor and rollback if that affects something on the JSI side.</li>
      <li>QEMU Agent. Provides some integration to PVE to help in the VM management.</li>
      <li>Install my public key for root login, helps with troubleshooting (I know, I’m not supposed to use root for this appliance, as I’m not supposed to deploy it con PVE/KVM either 🙂)</li>
    </ol>
  </li>
</ul>

<hr />
<p><strong>Update 1</strong>: I was contacted by someone that seems to be part of the JSI operations team, he asked nicely to rollback the hostname change since it breaks things for them.</p>

<p>As a long shot, I’m adding this as additional request for the PLM.</p>

<hr />

<p>Copy the OVA file to your PVE host (scp is your friend), and follow the procedure below.</p>

<ol>
  <li>First, we need to take note of the serial number in the two representations we’ll need.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># prepare the working directory</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> import/juniper
<span class="nb">cd </span>import/juniper
<span class="c"># at this point, copy the OVA file via SCP.</span>
<span class="c"># verify file existance</span>
<span class="nb">ls</span> <span class="nt">-l</span> vLWC-2.3.26-&lt;serial&gt;.ova
<span class="nt">-rw-r--r--</span> 1 ciro.iriarte ipausers 2466766848 Dec 19 17:01 vLWC-2.3.26-&lt;serial&gt;.ova
<span class="c"># We extract the content of the ZIP file</span>
<span class="nb">mkdir </span>vLWC-2.3.26-&lt;serial&gt;
<span class="nb">tar</span> <span class="nt">-C</span> vLWC-2.3.26-&lt;serial&gt;/ <span class="nt">-xf</span> vLWC-2.3.26-&lt;serial&gt;.ova
<span class="nb">cd </span>vLWC-2.3.26-&lt;serial&gt;/

<span class="c"># Recover the un-encrypted serial number from the filename and take note of the output of this command</span>
<span class="nb">basename</span> <span class="nt">-s</span> .ova <span class="si">$(</span><span class="nb">ls</span> <span class="nt">-1</span> ~/import/juniper/<span class="k">*</span>ova<span class="si">)</span>| <span class="nb">awk</span> <span class="nt">-F</span> <span class="s2">"-"</span> <span class="s1">'{ print $NF }'</span>

<span class="c"># Recover the encrypted serial number from the OVF file and take note of the output of this command</span>
<span class="nb">grep</span> <span class="nt">--color</span> serial_number ~/import/juniper/vLWC-2.3.26-&lt;serial&gt;/vLWC-2.3.26-&lt;serial&gt;.ovf |awk <span class="s1">'{ print $6 }'</span>|cut <span class="nt">-f</span> 2 <span class="nt">-d</span> <span class="s1">'"'</span></code></pre></figure>

<ol start="2">
  <li>We mount the system to modify it</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Make sure you start in the same directory we were working before. We move to the directory containing the disk image.</span>
<span class="nb">cd </span>vLWC-2.3.26-&lt;serial&gt;/

<span class="c"># Convert from VMDK to RAW</span>
qemu-img  convert vLWC-2.3.26-&lt;serial&gt;-disk1.vmdk vLWC-2.3.26-&lt;serial&gt;-disk1.raw

<span class="c"># Bind disk to loopback device</span>
losetup <span class="nt">-f</span> <span class="nt">-P</span> vLWC-2.3.26-&lt;serial&gt;-disk1.raw
<span class="c"># Confirm binding</span>
losetup <span class="nt">-l</span> 

<span class="c"># Create a mount point for the vLWC system</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/loop-root
<span class="c"># Mount the first partition</span>
<span class="nv">MYLOOP</span><span class="o">=</span><span class="si">$(</span>losetup <span class="nt">-l</span>|grep vLWC | <span class="nb">awk</span> <span class="s1">'{ print $1 }'</span><span class="si">)</span>
mount <span class="k">${</span><span class="nv">MYLOOP</span><span class="k">}</span>p1 /mnt/loop-root/
mount <span class="nt">-o</span> <span class="nb">bind</span> /dev /mnt/loop-root/dev/
mount <span class="nt">-o</span> <span class="nb">bind</span> /proc /mnt/loop-root/proc/
<span class="nb">chroot</span> /mnt/loop-root/</code></pre></figure>

<ol start="3">
  <li>As mentioned, we need to define the correct serial number. This would be the only mandatory change</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Define variables with serial data for your appliance</span>
<span class="nv">MYSERIAL</span><span class="o">=</span><span class="s2">"your plain text serial number"</span>
<span class="nv">MYSERIALENCRYPTED</span><span class="o">=</span><span class="s2">"your encrypted serial number"</span>

<span class="c"># Set encrypted serial parameter</span>
<span class="nb">echo</span> <span class="si">$(</span><span class="nb">cat</span> /opt/jsas/etc/vlwc-status.json | jq <span class="nt">--arg</span> val <span class="s2">"</span><span class="nv">$MYSERIALENCRYPTED</span><span class="s2">"</span> <span class="s1">'.properties.serial_number = $val'</span><span class="si">)</span> <span class="o">&gt;</span> /opt/jsas/etc/vlwc-status.json
<span class="c"># Set plain text serial</span>
<span class="nb">echo</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$MYSERIAL</span><span class="s2">"</span> <span class="o">&gt;</span> /home/jsas/nfxserialnum</code></pre></figure>

<ol start="4">
  <li>Nice to have changes. You make them while still <strong>inside</strong> the chroot environment. You could add all the block to a script if you wish.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">##</span>
<span class="c"># QEMU agent</span>
<span class="c">##</span>

<span class="c"># Temporary adjust name resolution to allow package installation from Internet</span>
<span class="nv">MYDNS</span><span class="o">=</span><span class="s2">"x.y.z.w"</span>
<span class="nb">mv</span> /etc/resolv.conf<span class="o">{</span>,.orig<span class="o">}</span>
<span class="nb">echo</span> <span class="s2">"nameserver </span><span class="nv">$MYDNS</span><span class="s2">"</span> <span class="o">&gt;</span> /etc/resolv.conf
<span class="c"># We install QEMU agent</span>
apt <span class="nb">install </span>qemu-guest-agent
<span class="c"># Restore original DNS resolution config</span>
<span class="nb">mv</span> /etc/resolv.conf<span class="o">{</span>.orig,<span class="o">}</span>

<span class="c">##</span>
<span class="c"># Hostname. We don't do this anymore until Juniper releases an update that moves the appliance version validation from the hostname.</span>
<span class="c">##</span>
<span class="c">#NO# MYHOSTNAME="short.TLD"</span>
<span class="c"># We backup original setup</span>
<span class="c">#NO# cp -p /etc/hostname{,.orig}</span>
<span class="c"># We define new hostname</span>
<span class="c">#NO# echo $MYHOSTNAME &gt; /etc/hostname</span>

<span class="c"># In case you did change the hostname, the rollback would be:</span>
<span class="c"># mv /etc/hostname{,.mine}</span>
<span class="c"># mv /etc/hostname{.orig,}</span>
<span class="c"># hostname $(cat /etc/hostname)</span>

<span class="c">##</span>
<span class="c"># Timezone</span>
<span class="c">##</span>

<span class="c"># Set your preferred timezone. You can get a list from timedatectl list-timezones</span>
timedatectl set-timezone &lt;Region&gt;/&lt;Country&gt;

<span class="c">##</span>
<span class="c"># SSH key for root</span>
<span class="c">##</span>

<span class="c"># Create the missing directory</span>
<span class="nb">mkdir</span> ~/.ssh
<span class="c"># Add your public key to the trusted keys</span>
<span class="nv">YOURPUBKEY</span><span class="o">=</span><span class="s2">"key-type your-full-public-key-should-go-here key-alias"</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$YOURPUBKEY</span><span class="s2">"</span> <span class="o">&gt;&gt;</span> ~/.ssh/authorized_keys</code></pre></figure>

<ol start="5">
  <li>We unmount the image and unbind the device</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># we exit the chroot</span>
<span class="nb">exit</span>
<span class="c"># umount filesystems</span>
umount /mnt/loop-root/dev /mnt/loop-root/proc /mnt/loop-root
<span class="c"># unbind device</span>
losetup <span class="nt">-d</span> <span class="nv">$MYLOOP</span></code></pre></figure>

<p>At this point, the disk image containes the setup required for your site/instance.</p>

<h2 id="vm-creation">VM creation</h2>

<p>For a vSphere supported solution, the OVF file is imported in vCloud Director or vCenter. It includes the VM definition (quantity &amp; type of CPU, RAM, disks &amp; network adapters).</p>

<p>We can’t use the OVF file for Proxmox VE, it should work, but in practice I found some formatting issues.</p>

<p>Login to your PVE machine via SSH and create a script to deploy the VM:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@pve01:~/import/juniper# <span class="nb">cat</span> <span class="o">&gt;</span> import-vlwc.sh<span class="o">&lt;&lt;</span><span class="sh">'</span><span class="no">EOF</span><span class="sh">'
#!/bin/bash
# We set the bridge &amp; the VLAN we need to integrate de VM to.
BRIDGE=vmbr0
VLAN=4089
# Storage destination for the appliance to live in
STORAGE=local-zfs
# Source disk image
DSK=vLWC-2.3.26-&lt;serial&gt;/vLWC-2.3.26-&lt;serial&gt;-disk1.raw

# We need the next VM ID available
NEWVM=</span><span class="si">$(</span>pvesh get /cluster/nextid<span class="si">)</span><span class="sh">
# Proper VM name
VMNAME=jsi-vlwc01.&lt;my-TLD&gt;

# We create the VM without disks. 6 vCPU + 16GB for up to 10k devices, most probably you can live with less resources.
qm create </span><span class="nv">$NEWVM</span><span class="sh"> --name </span><span class="nv">$VMNAME</span><span class="sh"> --bios seabios --machine q35 </span><span class="se">\</span><span class="sh">
--numa 1 --sockets 1 --cores 6 </span><span class="se">\</span><span class="sh">
--scsihw virtio-scsi-pci </span><span class="se">\</span><span class="sh">
--memory 16384 </span><span class="se">\</span><span class="sh">
--agent enabled=1

# We add the disk we have prepared, and attach it to the VM.
qm importdisk </span><span class="nv">$NEWVM</span><span class="sh"> </span><span class="nv">$DSK</span><span class="sh"> </span><span class="nv">$STORAGE</span><span class="sh">
qm set </span><span class="nv">$NEWVM</span><span class="sh"> -scsi0 </span><span class="k">${</span><span class="nv">DISK</span><span class="k">}</span><span class="sh"> </span><span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span><span class="sh">:vm-</span><span class="k">${</span><span class="nv">NEWVM</span><span class="k">}</span><span class="sh">-disk-0
# Boot device at BIOS
qm set </span><span class="nv">$NEWVM</span><span class="sh"> --boot order=scsi0

# We add NIC with the correct VLAN mapping
qm set </span><span class="nv">$NEWVM</span><span class="sh"> </span><span class="se">\</span><span class="sh">
--net0 model=virtio,bridge=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span><span class="sh">,firewall=1,tag=</span><span class="k">${</span><span class="nv">VLAN</span><span class="k">}</span><span class="sh">

# We start the VM
qm start
</span><span class="no">EOF

</span><span class="c"># Allow the script to be executed</span>
root@pve01:~/import/juniper# <span class="nb">chmod</span> +x import-vlwc.sh

<span class="c"># Create VM</span>
root@pve01:~/import/juniper# ./import-vlwc.sh</code></pre></figure>

<p>After a couple of minutes, you should be able to visit the web captive portal view through the appliance FQDN (assuming you have proper DHCP+DNS integration) or its IP address.</p>

<p>Use the plain text serial number to login and look for “Juniper Cloud Connected” + “Cloud Provisioned” to make sure it’s online.</p>

<h1 id="wishlist-for-the-plm">Wishlist for the PLM</h1>

<p>In case a Juniper vLWC appliance Product Owner passes by :), I would recommend the following to be included in the backlog:</p>

<h2 id="1--support-for-a-single-interface-setup">1- Support for a single interface setup</h2>

<p>I totally get the 3 interfaces setup for a physical appliance that’s colocated with a lone MX960 on a remote site forgotten by God. You get:</p>
<ul>
  <li>A “trusted” port that is either connected to a OOBM network serving several NEs or directly connected to the OOBM port on a lone router.</li>
  <li>An “outside” port that gives you access to Internet, probably comming from a NE in the ISP side of the house and shouln’t be trusted (no services like SSH listening).</li>
  <li>A “management” port that whould give access to an on-site engineer to the appliance, probably directly connected to his/her laptop.</li>
</ul>

<p>My uneducated guess is the virtual appliance basically followed a “shoehorn-in-a-vm whatever we had in the <a href="https://www.juniper.net/documentation/us/en/hardware/lwc-hardware-guide/lwc/topics/concept/lwc-description.html">physical appliance</a> version” approach.</p>

<p>A virtual machine most probably will be deployed at a cozy TIER-something datacenter in a proper virtualization cluster with a very elaborated logical setup for management &amp; monitoring, with L3 integrations to whatever destinations are required and services like filtering/segmentation outside of the appliance.</p>

<p>The ask would be: please support/allow a “single interface” setup as an alternative to the 3-port setup. Today it works as is, besides the EXT related errors in the status page.</p>

<h2 id="2--newer-linux-distribution">2- Newer Linux distribution</h2>

<p>As of vLWC-2.3.26, the appliance is based in Ubuntu 18.04.6 LTS which is already <a href="https://wiki.ubuntu.com/Releases">out of Standard Support after June 2023</a>. Only Ubuntu Pro support subscribers will get minimal patch releases, and after quick verification I can tell that’s not available on the deployed appliance (maybe even sporting a buggy openssl version?):</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@ggc-lnx:~# pro status
Failed to access URL: https://contracts.canonical.com/v1/resources?architecture<span class="o">=</span>amd64&amp;kernel<span class="o">=</span>4.15.0-212-generic&amp;series<span class="o">=</span>bionic&amp;virt<span class="o">=</span>kvm
Cannot verify certificate of server
Please check your openssl configuration.</code></pre></figure>

<p>Old Linux distributions instances and the security implications for a given environment is a threshed topic. I would just summarize the ask as: an OS refresh is due and Ubuntu 22.04.3 LTS is at your disposal. No, “we’ll eventually re-package a new version of the appliance with patches applied and you should re-deploy” is not acceptable</p>

<h2 id="3--support-for-pvekvm">3- Support for PVE/KVM</h2>

<p>Not all the Juniper shops run VMware vSphere on the compute side, and if they do, it may not always be the choice for each and every deployment. Even though I welcome the work done on the <a href="https://www.dmtf.org/standards/ovf">OVF</a> file preparation to automate the initial setup, I strongly recommend your team invests some time implementing support for <a href="https://cloudinit.readthedocs.io/en/latest/">cloud-init</a> in the initial setup script, which is the de facto standard to deploy virtual appliance in cloud environments (KVM on prem, vSphere, AWS, Azure, etc).</p>

<p>Some low hanging fruits at your disposal:</p>
<ul>
  <li>VirtIO network interface drivers are already available in the appliance (recommended for KVM/QEMU, not limited to VMXNET3). Used in this deployment.</li>
  <li>QEMU agent can easily be installed from Ubuntu repos pre-setup in the appliance (nice to have, covers some functionality of VMware Tools for KVM/QEMU). Used in this deployment.</li>
  <li>cloud-init agent can be easily installed from Ubuntu repos pre-setup in the appliance. Not used in this deployment.</li>
</ul>

<h2 id="4--allow-for-hostname-change">4- Allow for hostname change</h2>

<p>Due to some arguable requirements/conventions we have, we need to assign a proper hostname to the VM. It seems to be used for appliance versioning as noted a little early in the article.</p>
<ul>
  <li>We match the VM name &amp; OS hostname for inventory purposes (several tools data consolidation &amp; crosscheck).</li>
  <li>Part of SOX controls recommend proper hostname naming convention to be enforced.</li>
  <li>DHCP registration updates DNS records, having more than one appliance deployed messes up this automated integration.</li>
</ul>

<p>My strong recomendation is that you move appliance version reporting to a proper metadata entry somewhere else, and let hostname be a hostname.</p>]]></content><author><name>Ciro Iriarte</name></author><category term="Datacenter" /><category term="Networking" /><category term="Juniper" /><category term="QFX" /><category term="EX" /><category term="JunOS" /><summary type="html"><![CDATA[Walkthrough of analysis and deployment of a Juniper Virtual Lightweight Collector for "call home" functionality]]></summary></entry><entry><title type="html">Intermittent connection with Zerotier in Site-2-Site</title><link href="https://iriarte.it/sdn/networking/2023/12/09/intermittent-connection-with-zerotier-in-site-2-site.html" rel="alternate" type="text/html" title="Intermittent connection with Zerotier in Site-2-Site" /><published>2023-12-09T00:00:00-03:00</published><updated>2023-12-09T00:00:00-03:00</updated><id>https://iriarte.it/sdn/networking/2023/12/09/intermittent-connection-with-zerotier-in-site-2-site</id><content type="html" xml:base="https://iriarte.it/sdn/networking/2023/12/09/intermittent-connection-with-zerotier-in-site-2-site.html"><![CDATA[<h1 id="starting-point">Starting point</h1>

<p>I’ve been evaluating an idea to connect several sites using “just Internet”, versus traditional MPLS services. Out of the several options I have in mind, one of them is having a OPNSense as peering point for each LAN. All of them should be connected using Zerotier SDN in L3 with “managed routes”.</p>

<p>I’ve identified a 128 routes limit in the “managed routes” option, so probably will need to run a BGP route reflector in the final setup, but for the initial PoC managed routes would do.</p>

<!--more-->

<h2 id="basic-layout-of-the-poc">Basic layout of the PoC</h2>

<p>This is a basic diagram of what was tested. Each OPNSense endpoint would run the zerotier client and provide forwarding. Route information would be pushed by the controller.</p>

<figure>
  <a href="/assets/img/2023-12-09-zerotier-s2s-poc.png">
  <img src="/assets/img/2023-12-09-zerotier-s2s-poc.png" alt="network layout" />
  </a>
  <figcaption><i>Image 1 - PoC network layout </i></figcaption>
</figure>

<p>Routes as defined in the controller:</p>

<blockquote>
  <p>10.1.0.0/24 via 172.16.1.10</p>

  <p>10.2.0.0/24 via 172.16.1.20</p>
</blockquote>

<h1 id="the-issue">The issue</h1>

<p>During my tests, I’ve seen that the connection between sites (LAN on side A vs LAN on side B) is very intermittent, with 30-60% of packet lost.</p>

<h1 id="analysis-of-the-issue">Analysis of the issue</h1>

<p>I’ve created a small script for OPNsense to continuosly check peer status.</p>

<p>For your refence, file created as:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">mkdir</span> ~/bin
<span class="nb">touch</span> ~/bin/check-peer.csh
<span class="nb">chmod</span> +x ~/bin/check-peer.csh
vi ~/bin/check-peer.csh</code></pre></figure>

<p>Content to be inserted:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">#!/bin/csh</span>

<span class="c"># * Sat Dec  9 16:02:33 -03 2023 - ciro.iriarte (at) gmail.com</span>
<span class="c"># - Basic script to review peer underlay IP</span>

<span class="k">if</span>  <span class="o">(</span><span class="nv">$# </span><span class="o">!=</span> <span class="s2">"1"</span><span class="o">)</span> <span class="k">then
        </span><span class="nb">echo</span> <span class="s2">"usage:"</span>
        <span class="nb">echo</span> <span class="s2">"  </span><span class="nv">$0</span><span class="s2"> &lt;peer id&gt;"</span>
        <span class="nb">exit
</span>endif

<span class="k">while</span> <span class="o">(</span> 1 <span class="o">)</span>
  zerotier-cli peers | <span class="nb">grep</span> <span class="nv">$1</span>
  <span class="nb">sleep </span>3
end</code></pre></figure>

<p>Execution example:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@opnsense-A:~/bin <span class="c"># check-peer.sh 0f02bd4XXX</span>
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   252      413      &lt;public IP&gt;/25545
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      &lt;public IP&gt;/25545</code></pre></figure>

<p>What I found was that when the client had no response for the ICMP echo request (ping command), the output listed a change in IP for the peer to the internal LAN IP.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@opnsense-B:~/bin <span class="c"># check-peer.sh 0f02bd4XXX</span>
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   252      413      &lt;public IP&gt;/25545
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      &lt;public IP&gt;/25545
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      10.1.0.1/25645 
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      10.1.0.1/25645
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      10.1.0.1/25645
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      10.1.0.1/25645
0f02bd4XXX 1.12.2 LEAF       0 DIRECT   347      265      &lt;public IP&gt;/25545</code></pre></figure>

<p>In the past, I’ve seen through firewall logs and tcpdump captures that Zerotier is very promiscuous/aggressive and tries to reach peers using ANY available interface. I understand this is part of the “just works” experience they envisioned but it’s just odd it trying to reach an underlay peer through the overlay network (by means of L3 to a remote network in this case).</p>

<p>I would expect that Zerotier avoids using zt* interfaces by default.</p>

<h1 id="similar-reports">Similar reports</h1>

<h2 id="andy-liebman">Andy Liebman</h2>

<p>Andy Liebman reported the same scenario in the <a href="https://discuss.zerotier.com/t/troubleshooting-intermittent-connection-with-site-to-site/15435">forum</a>, but unluckily nobody provided feedback and by the time I reached the post, it was closed and no further comments were possible.</p>

<h2 id="ulegacyproblems-at-reddit">u/legacyproblems at Reddit</h2>

<p>Not necesarrily the same scenario, but it’s similar. This person blocked the unwanted flows through firewall rules:</p>

<blockquote>
  <p>I ran into a similar issue a while back where I have some devices acting as gateways onto my ZeroTier LAN and &gt; devices on the real LANs these gateways were connected to would tunnel through the same ZeroTier LAN via the &gt; gateways…</p>

  <p>Ended up adding a rule to block 9993 UDP on my ZeroTier LAN via the ZeroTier network rules engine.</p>
</blockquote>

<p><a href="https://www.reddit.com/r/zerotier/comments/15zw33a/how_to_make_zerotier_not_use_other_tunnels/">reference</a></p>

<h2 id="blacklists">Blacklists</h2>

<p>I recall seeing similar reports which got addressed through the use of <a href="https://docs.zerotier.com/config/#local-configuration-options">blacklists</a>, but can’t seem to find those posts right now to add links.</p>

<p>I prefer not going that route because with firewalls, adding new interfaces is kind of frequent and I will most probably forget about blacklisting thew new interfaces that are not supposed to be used for underlay traffic.</p>

<p>For my usecase whitelisting would make much more sense, telling Zerotier “use WAN1 &amp; WAN2 interfaces and nothing more”, but unluckily, that doesn’t seem to be supported.</p>

<h1 id="the-fix">The fix</h1>

<p>I’ve found that using the new <a href="https://docs.zerotier.com/multipath/">multipath</a> implementation seems to mimic the whitelist logic I’m after. I found this by accident, the private IP for the peer was only seen in the site that doesn’t have multipath configured.</p>

<p>Site A has two WAN interfaces. In the past, I’ve seen really weird browsing behaviour in OPNSense when having an active/active gateway group, that let me to fallback to failover groups. In this case, I would think Zerotier could actually work wonders in active/active scenarios but for the sake of simplicity, I kept the setup active/standby for Zerotier too.</p>

<p>local.conf would look like this:</p>

<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"settings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"defaultBondingPolicy"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rapid-active-backup"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"policies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"rapid-active-backup"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"basePolicy"</span><span class="p">:</span><span class="w"> </span><span class="s2">"active-backup"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"failoverInterval"</span><span class="p">:</span><span class="w"> </span><span class="mi">1000</span><span class="p">,</span><span class="w">
        </span><span class="nl">"links"</span><span class="p">:</span><span class="w">
        </span><span class="p">{</span><span class="w">
          </span><span class="nl">"ix0"</span><span class="p">:</span><span class="w">
          </span><span class="p">{</span><span class="w">
            </span><span class="nl">"ipvPref"</span><span class="p">:</span><span class="w"> </span><span class="mi">46</span><span class="p">,</span><span class="w">
            </span><span class="nl">"failoverTo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ix1"</span><span class="w">
          </span><span class="p">},</span><span class="w">
          </span><span class="nl">"ix1"</span><span class="p">:</span><span class="w">
          </span><span class="p">{</span><span class="w">
            </span><span class="nl">"ipvPref"</span><span class="p">:</span><span class="w"> </span><span class="mi">46</span><span class="p">,</span><span class="w">
            </span><span class="nl">"failoverTo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ix0"</span><span class="w">
          </span><span class="p">}</span><span class="w">
        </span><span class="p">}</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>

<p>Site B has one WAN interface, multipath conceptually doesn’t make sense in this scenario but still solves my “trying to reach the remote LAN through a peer in the remote LAN” issue.</p>

<p>local.conf would look like this:</p>

<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
  </span><span class="nl">"settings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"defaultBondingPolicy"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rapid-active-backup"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"policies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"rapid-active-backup"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"basePolicy"</span><span class="p">:</span><span class="w"> </span><span class="s2">"active-backup"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"failoverInterval"</span><span class="p">:</span><span class="w"> </span><span class="mi">1000</span><span class="p">,</span><span class="w">
        </span><span class="nl">"links"</span><span class="p">:</span><span class="w">
        </span><span class="p">{</span><span class="w">
          </span><span class="nl">"igb1"</span><span class="p">:</span><span class="w">
          </span><span class="p">{</span><span class="w">
            </span><span class="nl">"ipvPref"</span><span class="p">:</span><span class="w"> </span><span class="mi">46</span><span class="w">
          </span><span class="p">}</span><span class="w">
        </span><span class="p">}</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>

<p>I still see an sporadic entry for the LAN IP of the peer during my tests, but clients are reporting 0% of lost packets for a site to site ICMP test (ping?/pong!).</p>

<p>Will update the post if anything changes.</p>]]></content><author><name>Ciro Iriarte</name></author><category term="SDN" /><category term="Networking" /><category term="OPNsense" /><category term="Zerotier" /><summary type="html"><![CDATA[Found an odd behaviour with Zerotier while using OPNSense as L3 peering nodes for site 2 site integration.]]></summary></entry><entry><title type="html">Integrating Apstra Appliances Operating System to FreeIPA</title><link href="https://iriarte.it/homelab/sdn/2023/10/07/Integrating-Apstra-Appliances-OS-to-FreeIPA.html" rel="alternate" type="text/html" title="Integrating Apstra Appliances Operating System to FreeIPA" /><published>2023-10-07T00:00:00-03:00</published><updated>2023-10-07T00:00:00-03:00</updated><id>https://iriarte.it/homelab/sdn/2023/10/07/Integrating-Apstra-Appliances-OS-to-FreeIPA</id><content type="html" xml:base="https://iriarte.it/homelab/sdn/2023/10/07/Integrating-Apstra-Appliances-OS-to-FreeIPA.html"><![CDATA[<h1 id="intent">Intent</h1>

<p>If you’re working in any sane environment involving Unix/Linux servers, for sure you should have in place some kind of centralized authentication/authorization schema for operating system access. These days it’s LDAP or LDAP+Kerberos.</p>

<p>In my case, I’m using FreeIPA for the Linux ecosystem.</p>

<h1 id="apstra-appliances">Apstra appliances</h1>

<p>Apstra is configuration management &amp; monitoring tool for Datacenter Networks, I don’t think of it being a proper SDN controller (at least in the original term conception) because it’s more related to configuration management tools like Puppet or Ansible. Nonetheless, it’s part of the SDN tooling ecosystem.</p>

<p>Apstra appliances are based on Ubuntu LTS, gladly, we have the official FreeIPA accessible. Most probably, your Juniper representative will tell you that installing official Ubuntu packages is not supported.</p>

<p>I’ve implemented this with Apstra 4.1.0, 4.1.1 and 4.1.2 before, but 4.2.0 needs a tweak this time. It works and doesn’t interfere with its functionality in my experience, nonetheless, testing in a lab environment is recommended before moving to production. Proceed at your own risk.</p>

<h1 id="assumptions">Assumptions</h1>
<ol>
  <li>You have a properly deployed Apstra related ecosystems appliances (AOS, ZTP, workers)</li>
  <li>Correct Timezone is already setup</li>
  <li>NTP client configuration is in place</li>
  <li>DNS client configutarion is in place</li>
</ol>

<h1 id="freeipa-server-side-configuration">FreeIPA server side configuration</h1>

<p>Make sure the client has a DNS entry.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">ipa dnsrecord-add &lt;your domain&gt; &lt;client shortname&gt; <span class="nt">--a-rec</span> &lt;IPv4 <span class="k">for </span>the client&gt;
ipa dnsrecord-add &lt;your domain&gt; &lt;client shortname&gt; <span class="nt">--aaaa-rec</span> &lt;IPv6 <span class="k">for </span>the client&gt;</code></pre></figure>

<h1 id="python-libraries-and-aos">Python libraries and AOS</h1>

<p>Well, this is a special case with Apstra 4.2.0. There are python modules installed with <em>pip</em> outside of what apt based packages provide, they apparently are needed for the Apstra platform but break other tools like ipa-client-install for the AOS appliance (doesn’t happen with the ZTP appliance).</p>

<p>ZTP appliance:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">admin@apstra-ztp-03:~<span class="nv">$ </span>pip show cryptography
Name: cryptography
Version: 3.4.8
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The Python Cryptographic Authority and individual contributors
Author-email: cryptography-dev@python.org
License: BSD or Apache License, Version 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: ansible-core, paramiko</code></pre></figure>

<p>AOS appliance:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">admin@apstra-os-03:~<span class="nv">$ </span>pip show cryptography
Name: cryptography
Version: 38.0.0
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The Python Cryptographic Authority and individual contributors
Author-email: cryptography-dev@python.org
License: BSD-3-Clause OR Apache-2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: cffi
Required-by: paramiko</code></pre></figure>

<p>Note that the version for AOS is newer, and modules reside in /usr/local/lib instead of /usr/lib.</p>

<p>A container based application like Apstra should have everything needed for the application inside the container and should not require or mess host level libraries.</p>

<h2 id="how-it-affects-ipa-client-install">How it affects ipa-client-install</h2>
<p>The IPA configuration tool fails with this error:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">admin@apstra-os-03:~<span class="nv">$ </span>ipa-client-install 
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
  File <span class="s2">"/usr/sbin/ipa-client-install"</span>, line 22, <span class="k">in</span> &lt;module&gt;
    from ipaclient.install import ipa_client_install
  File <span class="s2">"/usr/lib/python3/dist-packages/ipaclient/install/ipa_client_install.py"</span>, line 7, <span class="k">in</span> &lt;module&gt;
    from ipaclient.install import client
  File <span class="s2">"/usr/lib/python3/dist-packages/ipaclient/install/client.py"</span>, line 37, <span class="k">in</span> &lt;module&gt;
    from ipalib import api, errors, x509
  File <span class="s2">"/usr/lib/python3/dist-packages/ipalib/__init__.py"</span>, line 921, <span class="k">in</span> &lt;module&gt;
    from ipalib.frontend import Command, LocalOrRemote, Updater
  File <span class="s2">"/usr/lib/python3/dist-packages/ipalib/frontend.py"</span>, line 31, <span class="k">in</span> &lt;module&gt;
    from ipalib.parameters import create_param, Param, Str, Flag
  File <span class="s2">"/usr/lib/python3/dist-packages/ipalib/parameters.py"</span>, line 125, <span class="k">in</span> &lt;module&gt;
    from ipalib.x509 import <span class="o">(</span>
  File <span class="s2">"/usr/lib/python3/dist-packages/ipalib/x509.py"</span>, line 91, <span class="k">in</span> &lt;module&gt;
    @crypto_utils.register_interface<span class="o">(</span>crypto_x509.Certificate<span class="o">)</span>
<span class="k">**</span>AttributeError: module <span class="s1">'cryptography.utils'</span> has no attribute <span class="s1">'register_interface'</span><span class="nb">.</span> Did you mean: <span class="s1">'verify_interface'</span>?<span class="k">**</span></code></pre></figure>

<p>It seems FreeIPA was using a function that was not expected to be used outside of the capy/cryptography implementation. The latter project team <a href="https://github.com/pyca/cryptography/pull/7234">decided to deprecate the function</a> in 38.0, installing the newer library outside of what the distro provides, breaks the FreeIPA client provided by it.</p>

<p>FreeIPA project is <a href="https://github.com/freeipa/freeipa/pull/6455">complying with the upstream change</a>, and should be included in 4.11+</p>

<h2 id="the-correct-way-of-dealing-with-this">The correct way of dealing with this</h2>

<h3 id="golden-rule">Golden rule</h3>

<p>You don’t mess with Linux Distribution provided libraries. Ever.</p>

<h3 id="tools-at-hand-for-software-packagers">Tools at hand for software packagers</h3>

<p>Containers is one current software delivery solution that could address this. In fact, Juniper is using docker containers. I’m a little lost about why this is a problem today with the Apstra AOS appliance, but packaging all the dependancies within the containers should be the correct course of action.</p>

<h2 id="workaround-for-juniper-customers">Workaround for Juniper customers</h2>

<p>All we can do until Juniper properly packages Apstra as a set of container images is to choose the less ugly route. In this case, I’ve rebuilt a package from Ubuntu Ubuntu 23.04, which is based on FreeIPA 4.9.11 source code, for Ubuntu 22.04 (in which the Apstra appliance is based).</p>

<p>We’ll add the repository just for AOS, ZTP appliance can use the stock packages.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo "deb http://download.opensuse.org/repositories/home:/ciriarte:/apstra-4.2.0-fix/xUbuntu_22.04/ /" | sudo tee /etc/apt/sources.list.d/home:ciriarte:apstra-4.2.0-fix.list'</span>
<span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'curl -fsSL https://download.opensuse.org/repositories/home:ciriarte:apstra-4.2.0-fix/xUbuntu_22.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_ciriarte_apstra-4.2.0-fix.gpg &gt; /dev/null'</span>
<span class="nb">sudo chmod </span>a+r /etc/apt/trusted.gpg.d/home_ciriarte_apstra-4.2.0-fix.gpg
<span class="nb">sudo chmod </span>a+r /etc/apt/sources.list.d/home:ciriarte:apstra-4.2.0-fix.list
<span class="nb">sudo </span>apt update</code></pre></figure>

<p>After adding the repository, just follow the below procedure.</p>

<h1 id="clients-configuration">Clients configuration</h1>

<p>For each client in question, ZTP appliance, main AOS appliance or workers you would need to execute these steps.</p>

<h2 id="dns-validation">DNS validation</h2>
<p>DNS is critical in this environment, make sure the clients can actually locate your IP servers.</p>

<ol>
  <li>Test the the client is actually configured</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">dig _ldap._tcp.&lt;your domain&gt; SRV</code></pre></figure>

<ol start="2">
  <li>Test that all expected servers are actually answering. There are some funky scenarios were some servers answer and others dont.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">dig @&lt;recursor/resolver 1 IP&gt;  _ldap._tcp.&lt;your domain&gt; SRV
dig @&lt;recursor/resolver 2 IP&gt;  _ldap._tcp.&lt;your domain&gt; SRV</code></pre></figure>

<h2 id="client-installation--configuration">Client installation &amp; configuration</h2>

<p>The FreeIPA client is part of the default repositories, just install it with the package manager:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt <span class="nb">install</span> <span class="nt">-y</span> freeipa-client
<span class="nb">sudo </span>ipa-client-install <span class="nt">--mkhomedir</span> <span class="nt">--enable-dns-updates</span> <span class="se">\</span>
 <span class="nt">--domain</span><span class="o">=</span>&lt;IPA domain <span class="k">in </span>lowercase&gt; <span class="nt">--realm</span><span class="o">=</span>&lt;IPA domain <span class="k">in </span>uppercase&gt;</code></pre></figure>

<h2 id="limit-ssh-access">Limit SSH access</h2>

<p>It’s good practice to limit access via SSH to users that really need access to the node.</p>

<p>Thinking about this, it should be done with FreeIPA’s HBAC module + pam_sss.so. I can’t recall why in this environment we’re hardcoding the group at the sshd level, will keep it for consistency. <strong>To be revisited</strong></p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo "AllowGroups ssh-allow gapstra-administrator" &gt;&gt; /etc/ssh/sshd_config'</span>
<span class="nb">sudo </span>systemctl restart sshd</code></pre></figure>]]></content><author><name>Ciro Iriarte</name></author><category term="HomeLab" /><category term="SDN" /><category term="Apstra" /><category term="FreeIPA" /><category term="Ubuntu" /><summary type="html"><![CDATA[IPA client installation on Ubuntu based appliances for Apstra 4.2.0]]></summary></entry><entry><title type="html">Deprecated GPG Key in OpenBuild Service</title><link href="https://iriarte.it/homelab/2023/10/07/deprecated-gpg-key-in-openbuild-service.html" rel="alternate" type="text/html" title="Deprecated GPG Key in OpenBuild Service" /><published>2023-10-07T00:00:00-03:00</published><updated>2023-10-07T00:00:00-03:00</updated><id>https://iriarte.it/homelab/2023/10/07/deprecated-gpg-key-in-openbuild-service</id><content type="html" xml:base="https://iriarte.it/homelab/2023/10/07/deprecated-gpg-key-in-openbuild-service.html"><![CDATA[<h1 id="openbuild-service">OpenBuild Service</h1>

<p>The openSuSE guys built a nice selfservice Linux package construction solution, which I believe was originally named openSuSE Build Service (OBS). Later it was renamed to OpenBuild Service to reflect the capability of building packages for more distros than just openSuSE.</p>

<p>More info: <a href="https://en.wikipedia.org/wiki/Open_Build_Service">Wikipedia</a></p>

<h1 id="gpg-signature">GPG Signature</h1>

<p>To protect the each Linux distribution community, the creators of each package management solution thought about signing the repository contents and packages with GPG keys. It was very intelligent and very useful as a basic hygienic/safety measure.</p>

<p>Such GPG signature functionality is included in OBS.</p>

<h1 id="dsa-1024-key-deprecation">DSA-1024 key deprecation</h1>

<p>As many other software packaging communities, Ubuntu deprecated DSA-1024 based GPG keys around 2016 which means apt doesn’t accept those keys anymore.</p>

<h1 id="the-issue">The issue</h1>

<p>In my case, I created a sub-project for Ubuntu 22.04 packages. Given I already had a parent project with a DSA-1024 key that was inherited by the new sub-project, all the packages were signed by it and later the package manager would complain about the key without too much feedback regarding why.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">Err:8 http://download.opensuse.org/repositories/home:/ciriarte:/apstra-4.2.0-fix/xUbuntu_22.04  InRelease
  The following signatures were invalid: 8FE63E73E027DE951D5EAFDBA6F3579ABB33D482
Reading package lists... Done
W: GPG error: http://download.opensuse.org/repositories/home:/ciriarte:/apstra-4.2.0-fix/xUbuntu_22.04  InRelease: The following signatures were invalid: 8FE63E73E027DE951D5EAFDBA6F3579ABB33D482
E: The repository <span class="s1">'http://download.opensuse.org/repositories/home:/ciriarte:/apstra-4.2.0-fix/xUbuntu_22.04  InRelease'</span> is not signed.
N: Updating from such a repository can<span class="s1">'t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.</span></code></pre></figure>

<p>Usually you would see a message stating missing key, or not accepted cipher suite, or something along the lines. In this case, it was a generic “The following signatures were invalid” message.</p>

<p>Taking a look at the installed public keys, I spotted my repository was using a dsa1024 key, while all the rest used rsa4096.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">admin@apstra-os-03:~<span class="nv">$ </span><span class="nb">sudo </span>apt-key list
<span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>admin:
Warning: apt-key is deprecated. Manage keyring files <span class="k">in </span>trusted.gpg.d
instead <span class="o">(</span>see apt-key<span class="o">(</span>8<span class="o">))</span><span class="nb">.</span>
/etc/apt/trusted.gpg
<span class="nt">--------------------</span>
pub   rsa1024 2009-01-26 <span class="o">[</span>SC]
      3AE6 3EC8 2014 45CB 55E6  B1EE DCCB 2270 E771 6B13
uid           <span class="o">[</span> unknown] Launchpad PPA <span class="k">for </span>Stéphane Graber

pub   rsa4096 2017-02-22 <span class="o">[</span>SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           <span class="o">[</span> unknown] Docker Release <span class="o">(</span>CE deb<span class="o">)</span> &lt;docker@docker.com&gt;
sub   rsa4096 2017-02-22 <span class="o">[</span>S]

/etc/apt/trusted.gpg.d/home_ciriarte_apstra-4.2.0-fix.gpg
<span class="nt">---------------------------------------------------------</span>
pub   dsa1024 2008-01-22 <span class="o">[</span>SC] <span class="o">[</span>expires: 2025-12-13]
      8FE6 3E73 E027 DE95 1D5E  AFDB A6F3 579A BB33 D482
uid           <span class="o">[</span> unknown] home:ciriarte OBS Project
&lt;home:ciriarte@build.opensuse.org&gt;

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
<span class="nt">------------------------------------------------------</span>
pub   rsa4096 2012-05-11 <span class="o">[</span>SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           <span class="o">[</span> unknown] Ubuntu CD Image Automatic Signing Key <span class="o">(</span>2012<span class="o">)</span>
&lt;cdimage@ubuntu.com&gt;

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
<span class="nt">------------------------------------------------------</span>
pub   rsa4096 2018-09-17 <span class="o">[</span>SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           <span class="o">[</span> unknown] Ubuntu Archive Automatic Signing Key <span class="o">(</span>2018<span class="o">)</span>
&lt;ftpmaster@ubuntu.com&gt;</code></pre></figure>

<h1 id="the-fix">The fix</h1>

<p>In the end, the one to blame was the old deprecated key type (my home project was created a long time ago). I couln’t find a way to create a new GPG key in the <a href="https://build.opensuse.org">web UI</a>, either it’s not there or I’m just blind.</p>

<p>We’ll be using the OBS CLI client/tool: osc.</p>

<p>In the osc man page we can find:</p>

<blockquote>
  <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>   signkey
         Manage Project Signing Key

         osc signkey [--create|--delete|--extend] &lt;PROJECT&gt; osc signkey [--notraverse] &lt;PROJECT&gt;

         This command is for managing gpg keys. It shows the public key by default. There is no way to download or upload the private  part  of  a
         key by design.

         However you can create a new own key. You may want to consider to sign the public key with your own existing key.

         If  a  project  has  no  key,  the  key  from upper level project will be used (e.g. when dropping "KDE:KDE4:Community" key, the one from
         "KDE:KDE4" will be used).

         WARNING: THE OLD KEY CANNOT BE RESTORED AFTER USING DELETE OR CREATE

         Usage:
             osc signkey [ARGS...]

         Options:
             -h, --help  show this help message and exit
             --sslcert   fetch SSL certificate instead of GPG key
             --notraverse
                         don't traverse projects upwards to find key
             --delete    delete the gpg signing key in this project
             --extend    extend expiration date of the gpg public key for this
                         project
             --create    create new gpg signing key for this project
</code></pre></div>  </div>
</blockquote>

<p>In a Linux machine, you need to follow this procedure:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># Install the tool via your favorite package manager</span>
<span class="nb">sudo </span>zypper <span class="k">in</span> <span class="nt">-y</span> osc

<span class="c"># First execution will ask for OBS credentials</span>
osc

<span class="c"># We query the current key for the repository</span>
osc signkey home:ciriarte:apstra-4.2.0-fix

<span class="c"># We create a new key for the repository</span>
osc signkey <span class="nt">--create</span> home:ciriarte:apstra-4.2.0-fix

<span class="c"># We query the new key for the repository (should be different if the creation worked)</span>
osc signkey home:ciriarte:apstra-4.2.0-fix

<span class="c"># We wipe all the binaries to force recreation and resign of the packages with the new key</span>
osc wipebinaries <span class="nt">--all</span> home:ciriarte:apstra-4.2.0-fix freeipa</code></pre></figure>

<p>On the client, after reimporting the public key, you should see it’s not a dsa1024 anymore.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@apstra-os-03:~# apt-key list
Warning: apt-key is deprecated. Manage keyring files <span class="k">in </span>trusted.gpg.d instead <span class="o">(</span>see apt-key<span class="o">(</span>8<span class="o">))</span><span class="nb">.</span>
/etc/apt/trusted.gpg
<span class="nt">--------------------</span>
pub   rsa1024 2009-01-26 <span class="o">[</span>SC]
      3AE6 3EC8 2014 45CB 55E6  B1EE DCCB 2270 E771 6B13
uid           <span class="o">[</span> unknown] Launchpad PPA <span class="k">for </span>Stéphane Graber

pub   rsa4096 2017-02-22 <span class="o">[</span>SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           <span class="o">[</span> unknown] Docker Release <span class="o">(</span>CE deb<span class="o">)</span> &lt;docker@docker.com&gt;
sub   rsa4096 2017-02-22 <span class="o">[</span>S]

/etc/apt/trusted.gpg.d/home_ciriarte_apstra-4.2.0-fix.gpg
<span class="nt">---------------------------------------------------------</span>
pub   rsa4096 2023-10-07 <span class="o">[</span>SC] <span class="o">[</span>expires: 2025-12-15]
      37C6 811B 46C8 134C 5705  8A45 33E6 7F5C 0370 8C89
uid           <span class="o">[</span> unknown] home:ciriarte:apstra-4.2.0-fix OBS Project &lt;home:ciriarte:apstra-4.2.0-fix@build.opensuse.org&gt;

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
<span class="nt">------------------------------------------------------</span>
pub   rsa4096 2012-05-11 <span class="o">[</span>SC]</code></pre></figure>]]></content><author><name>Ciro Iriarte</name></author><category term="HomeLab" /><category term="OpenBuildService" /><category term="openSuSE" /><category term="Ubuntu" /><summary type="html"><![CDATA[Quick fix to force key replacement]]></summary></entry><entry><title type="html">Deploying Apstra 4.2.0 on Proxmox VE</title><link href="https://iriarte.it/homelab/sdn/2023/10/03/deploying-apstra-4.2.0-on-PVE.html" rel="alternate" type="text/html" title="Deploying Apstra 4.2.0 on Proxmox VE" /><published>2023-10-03T00:00:00-03:00</published><updated>2023-10-03T00:00:00-03:00</updated><id>https://iriarte.it/homelab/sdn/2023/10/03/deploying-apstra-4.2.0-on-PVE</id><content type="html" xml:base="https://iriarte.it/homelab/sdn/2023/10/03/deploying-apstra-4.2.0-on-PVE.html"><![CDATA[<h1 id="apstra-appliances">Apstra appliances</h1>

<p>Juniper is not distributing appliances per se for KVM deployments, but OS disk images in QCOW2 format:</p>

<p><strong>aos_server_4.2.0-236.qcow2</strong></p>

<p>Main Apstra server appliance, also used for optional workers for the scaleout setup: you use the same image for the worker nodes (didn’t find this mentioned in the documentation but was mentioned by a Juniper guy)</p>

<p><strong>apstra-ztp-4.2.0-34.qcow2</strong></p>

<p>Zero Touch provisioning appliance. This is an optional ZTP appliance running PXE+TFTP. It can be used as pure TFTP with a pre-existing DHCP server.</p>

<h1 id="installation">Installation</h1>

<h2 id="overview">Overview</h2>

<p>For the VM sizing, the vendor has a <a href="https://www.juniper.net/documentation/us/en/software/apstra4.2/apstra-install-upgrade/topics/ref/apstra-server-resources.html">recommendated sizing table</a>:</p>

<table>
  <tbody>
    <tr>
      <td>Resource</td>
      <td>Value</td>
    </tr>
    <tr>
      <td>vCPU</td>
      <td>8</td>
    </tr>
    <tr>
      <td>RAM</td>
      <td>64GB + 500MB per installed offbox agent</td>
    </tr>
    <tr>
      <td>Disk</td>
      <td>160GB</td>
    </tr>
  </tbody>
</table>

<p>In my experience, that many resources are not needed, especially the RAM. For my test environment, I’ll slash RAM to 32GB and most probably that’s not supported, I can live with that, YMMV.</p>

<p>We’ll create the empty virtual machines in Proxmox VE 7.4-15, importing afterwards the QCOW2 disks as needed. The procedure asumes you have downloaded all the files to a working directory (curl/wget are your friends) in a PVE node.</p>

<h2 id="apstra-os">Apstra OS</h2>

<h3 id="vm-creation">VM creation</h3>

<p>We’ll create a virtual machine with CLI, with 4 vCPUs and 32GB of RAM.</p>

<p>The network is being served via the default bridge and storage via local ZFS repository. Options are self explanatory, adjust as needed.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">VMNAME</span><span class="o">=</span>apstra-os-03.ipa.&lt;my TLD&gt;
<span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>pvesh get /cluster/nextid<span class="si">)</span>
<span class="nv">STORAGE</span><span class="o">=</span>local-zfs
<span class="nv">VCPU</span><span class="o">=</span>4
<span class="nv">VRAM</span><span class="o">=</span>32768
<span class="nv">VLANLAB</span><span class="o">=</span>&lt;my vlan&gt;
<span class="nv">BRIDGE</span><span class="o">=</span>vmbr0

<span class="c"># OS doesn't support UEFI</span>
qm create <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--bios</span> seabios <span class="nt">--machine</span> q35 <span class="se">\</span>
<span class="nt">--ostype</span> l26 <span class="nt">--agent</span> 1 <span class="se">\</span>
<span class="nt">--sockets</span> 1 <span class="nt">--cores</span> <span class="k">${</span><span class="nv">VCPU</span><span class="k">}</span> <span class="nt">--cpu</span> <span class="nv">cputype</span><span class="o">=</span>kvm64 <span class="se">\</span>
<span class="nt">--scsihw</span> virtio-scsi-single <span class="se">\</span>
<span class="nt">--memory</span> <span class="k">${</span><span class="nv">VRAM</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--net0</span> <span class="nv">model</span><span class="o">=</span>virtio,bridge<span class="o">=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span>,firewall<span class="o">=</span>0,tag<span class="o">=</span><span class="k">${</span><span class="nv">VLANLAB</span><span class="k">}</span></code></pre></figure>

<h3 id="importing-disk">Importing disk</h3>

<p>Now, we’ll be importing the QCOW2 file provided by Juniper.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">VMNAME</span><span class="o">=</span>apstra-os-03.ipa.&lt;my TLD&gt;
<span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>qm list|grep <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span>|awk <span class="s1">'{ print $1 }'</span><span class="si">)</span>
<span class="nv">STORAGE</span><span class="o">=</span>local-zfs

qm disk import <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> aos_server_4.2.0-236.qcow2  <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">-scsi0</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> <span class="nv">order</span><span class="o">=</span>scsi0</code></pre></figure>

<h3 id="additional-disk">Additional disk</h3>

<p>According to the documentation, the OS disk is 80GB only and optionally you could add a second 80GB. Given it’s not onerous and I won’t be using additional worker nodes, I’ll add the second disk.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">VMNAME</span><span class="o">=</span>apstra-os-03.ipa.&lt;my TLD&gt;
<span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>qm list|grep <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span>|awk <span class="s1">'{ print $1 }'</span><span class="si">)</span>
<span class="nv">STORAGE</span><span class="o">=</span>local-zfs

qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--scsi1</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:80
<span class="c"># we start the VM</span>
qm start <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span></code></pre></figure>

<h2 id="ztp-appliance">ZTP appliance</h2>

<p>DHCP &amp; TFTP are not that demanding, you can go with 2 vCPUs and 4GB of RAM.</p>

<p>The network is being served via the default bridge and storage via local ZFS repository. Options are self explanatory, adjust as needed.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">VMNAME</span><span class="o">=</span>apstra-ztp-03.ipa.&lt;my TLD&gt;
<span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>pvesh get /cluster/nextid<span class="si">)</span>
<span class="nv">STORAGE</span><span class="o">=</span>local-zfs
<span class="nv">VCPU</span><span class="o">=</span>2
<span class="nv">VRAM</span><span class="o">=</span>4096
<span class="nv">VLANLAB</span><span class="o">=</span>&lt;my vlan&gt;
<span class="nv">BRIDGE</span><span class="o">=</span>vmbr0

<span class="c"># OS doesn't support UEFI</span>
qm create <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--name</span> <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--bios</span> seabios <span class="nt">--machine</span> q35 <span class="se">\</span>
<span class="nt">--ostype</span> l26 <span class="nt">--agent</span> 1 <span class="se">\</span>
<span class="nt">--sockets</span> 1 <span class="nt">--cores</span> <span class="k">${</span><span class="nv">VCPU</span><span class="k">}</span> <span class="nt">--cpu</span> <span class="nv">cputype</span><span class="o">=</span>kvm64 <span class="se">\</span>
<span class="nt">--scsihw</span> virtio-scsi-single <span class="se">\</span>
<span class="nt">--memory</span> <span class="k">${</span><span class="nv">VRAM</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--net0</span> <span class="nv">model</span><span class="o">=</span>virtio,bridge<span class="o">=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span>,firewall<span class="o">=</span>0,tag<span class="o">=</span><span class="k">${</span><span class="nv">VLANLAB</span><span class="k">}</span></code></pre></figure>

<h3 id="importing-disk-1">Importing disk</h3>

<p>Now, we’ll be importing the QCOW2 file provided by Juniper.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">VMNAME</span><span class="o">=</span>apstra-ztp-03.ipa.&lt;my TLD&gt;
<span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>qm list|grep <span class="k">${</span><span class="nv">VMNAME</span><span class="k">}</span>|awk <span class="s1">'{ print $1 }'</span><span class="si">)</span>
<span class="nv">STORAGE</span><span class="o">=</span>local-zfs

qm disk import <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> apstra-ztp-4.2.0-34.qcow2  <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">-scsi0</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:vm-<span class="k">${</span><span class="nv">VMID</span><span class="k">}</span><span class="nt">-disk-1</span>
qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--boot</span> <span class="nv">order</span><span class="o">=</span>scsi0
qm start <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span></code></pre></figure>

<h1 id="initial-configuration">Initial Configuration</h1>
<h2 id="aos-setup">AOS setup</h2>

<ol>
  <li>Search for the IP of the appliance in your DHCP leases and connect through SSH with admin/admin.</li>
  <li>On first login set new password as requested.</li>
  <li>On the forced assistant, also set the Apstra UI password as proposed. Cancel at the main menu after setting up the password.</li>
  <li>Change hostname to something that makes sense for your environment. With admin user run:</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>aos_hostname apstra-os-03.ipa.&lt;my TLD&gt;</code></pre></figure>

<ol start="5">
  <li>Adjust timezone to match your environment.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>timedatectl set-timezone &lt;my TZ&gt;</code></pre></figure>

<ol start="6">
  <li>As admin, update the operating system. Juniper most probably will tell you that this is not supported. The only issue I had in the past was 4.1.0 breaking because there was a system-wide Juniper library that broke a containerized application (go figure) and required manual dependancies update to fix it. It has been flawless for 4.1.2 and now 4.2.0, pam.d files will be mentioned, keep the locally modified files.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt update
<span class="nb">sudo </span>apt upgrade <span class="nt">-y</span></code></pre></figure>

<ol start="7">
  <li>Install QEMU guest agent</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt <span class="nb">install</span> <span class="nt">-y</span> qemu-guest-agent</code></pre></figure>

<ol start="8">
  <li>Reboot after full system update
sudo shutdown -r now</li>
</ol>

<h2 id="ztp-appliance-setup">ZTP appliance setup</h2>

<ol>
  <li>Search for the IP of the appliance in your DHCP leases and connect through SSH with admin/admin.</li>
  <li>Set hostname</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>hostnamectl set-hostname apstra-ztp-03.ipa.&lt;my TLD&gt;
<span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo "127.0.0.1 apstra-ztp-03.ipa.&lt;my TLD&gt;" &gt;&gt; /etc/hosts'</span></code></pre></figure>

<ol start="3">
  <li>Adjust timezone to match your environment.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>timedatectl set-timezone &lt;my TZ&gt;</code></pre></figure>

<ol start="4">
  <li>Clean messed-up config structure: For some reason, the Juniper guys thought it was a good idea to create a directory where a file should be and that breaks Ubuntu upgrade process. We clean that up (what gives Juniper?!)</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo rmdir</span> /etc/udev/rules.d/70-persistent-net.rules</code></pre></figure>

<ol start="5">
  <li>Update operating system. Again, probably Juniper will tell you it’s not supported.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt update
<span class="nb">sudo </span>apt upgrade <span class="nt">-y</span></code></pre></figure>

<ol start="6">
  <li>At the Apstra Web UI, create a user with device_ztp role. We’ll use it in the next step.</li>
  <li>Back to the ZTP appliance, configure the application to report to the AOS server. For some reason, it doesn’t work with FQDN but only with IP.</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">cat</span> <span class="o">&gt;</span> /containers_data/status/app/aos.conf <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh">
{
   "ip": "&lt;your-aos-ip&gt;",
   "user": "&lt;your ztp user&gt;",
   "password": "&lt;the super secure password&gt;"
}
EOF</span></code></pre></figure>

<ol start="8">
  <li>Install QEMU guest agent</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>apt <span class="nb">install</span> <span class="nt">-y</span> qemu-guest-agent</code></pre></figure>

<ol start="9">
  <li>Reboot virtual machine</li>
</ol>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">sudo </span>shutdown <span class="nt">-r</span> now</code></pre></figure>

<h1 id="migration-from-412">Migration from 4.1.2</h1>

<p>If you already have a working environment, most probably you would like to migrate all the configuration to your new installation, there is a script you should run in the new Apstra Server.</p>

<p>Take into account that the script will fail if there are any uncommited changes or if devices are not reachable. Make sure the running state of the source environment is clean (no errors).</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">aos_import_state <span class="nt">--ip-address</span> apstra-os-02.ipa.&lt;my TLD&gt; <span class="nt">--username</span> admin</code></pre></figure>

<p>Note:</p>
<ul>
  <li>The configuration migration would also replace the web UI password with the configuration from the original server.</li>
</ul>]]></content><author><name>Ciro Iriarte</name></author><category term="HomeLab" /><category term="SDN" /><category term="Apstra" /><category term="Proxmox" /><category term="Virtualization" /><summary type="html"><![CDATA[Step by step deployment of Apstra 4.2.0 AOS & ZTP appliances]]></summary></entry><entry><title type="html">vSAN ESA on ESXi-over-PVE</title><link href="https://iriarte.it/homelab/2023/09/07/vsan-esa-on-esxi-over-pve.html" rel="alternate" type="text/html" title="vSAN ESA on ESXi-over-PVE" /><published>2023-09-07T00:00:00-04:00</published><updated>2023-09-07T00:00:00-04:00</updated><id>https://iriarte.it/homelab/2023/09/07/vsan-esa-on-esxi-over-pve</id><content type="html" xml:base="https://iriarte.it/homelab/2023/09/07/vsan-esa-on-esxi-over-pve.html"><![CDATA[<h1 id="a-little-bit-of-context">A little bit of context</h1>

<h2 id="vsan-osa">vSAN OSA</h2>
<p>One of my biggest griefs as a customer regarding the VMware SDDC offering is vSAN OSA. When I first saw it in action, it looked like a “me too” response to <a href="https://www.nutanix.com/mx/info/software-defined-storage">Nutanix</a>, which we introduced successfully I believe as a first customer in Paraguay during 2016-ish (if memory serves right).</p>

<p>Also, coming from the background of FibreChannel based solutions like <a href="https://www.youtube.com/watch?v=Zp3yMGlzZhM">EMC’s vMAX</a> &amp; <a href="https://knowledge.hitachivantara.com/Documents/Storage/VSP_G1X00_and_VSP_F1500/80-06-6x/Hardware_Guide/03_Hardware_architecture">Hitachi’s VSP</a> it was a really hard sell feature wise (no snapshots, no tiering, no replication, etc) and going with 20 nodes clusters just because we needed capacity for big fat Oracle Databases didn’t make sense (it’s not cost effective)</p>

<p>Another way of seeing it, if you compare vSAN OSA on vSphere to other solutions serving Xen or KVM, it lacks flexibility:</p>
<ul>
  <li>single pool per cluster</li>
  <li>single disk type</li>
  <li>cache drive as important failure points</li>
</ul>

<p>When non-ESXi at sight, we would just go with any other alternatives like <a href="https://docs.ceph.com/en/latest/architecture/">CEPH</a> or <a href="https://moosefs.com/blog/architecture/">MooseFS</a> (no <a href="https://docs.gluster.org/en/main/Quick-Start-Guide/Architecture/">GlusterFS</a>, nobody likes you….)</p>

<p>On the bright side, it’s an interesting entry point to better alternatives for folks behind technologically: think iSCSI with 1GbE or CIFS/SMB3 served from your Windows Server file server :s</p>

<p>All in all, it makes sense if:</p>
<ul>
  <li>you’re a VMware shop with very limited infra team(s)</li>
  <li>you want a quick way to introduce a virtualization platform</li>
  <li>there’s no centralized Storage + SAN in place to start with</li>
  <li>your storage needs are not disproportionate in comparison with compute</li>
  <li>you don’t need advanced storage services like taking disks snapshots from your 40TB live production Oracle database (10*4TB + ASM) to present them to a secondary reporting instance.</li>
</ul>

<h2 id="vsan-esa">vSAN ESA</h2>
<p>First time I heard about <a href="https://core.vmware.com/blog/introduction-vsan-express-storage-architecture">vSAN ESA</a> was in the 2022 VMware Explore event. It’s not perfect, but it’s a step in the right direction.</p>

<p>Just to be clear, I’m quite happy with the evolution. :)</p>

<p>Make sure you watch the <a href="https://www.vmware.com/explore/video-library/video-landing.html?sessionid=1663909090011001QAvr&amp;videoId=6315818080112">Get to Know the Next-Generation of vSAN Architecture</a> on demand VMware Explore 2022 session recording to understand all the goodies.</p>

<h1 id="going-back-to-our-proxmox-nested-environment">Going back to our Proxmox nested environment.</h1>

<p>With that context, I really recommend you to explore the changes in the new architecture.</p>

<p>If you would like to get your feet wet, vSAN ESA can be deployed in our nested ESX-on-KVM/Proxmox environment.</p>

<p>Of course:</p>
<ul>
  <li>it won’t be supported, but will allow you to play with it</li>
  <li>actual performance tests will require a property supported physical setup</li>
</ul>

<p>Warnings will be present, because we don’t meet any <a href="https://kb.vmware.com/s/article/90343">supported configuration</a>, remarkably:</p>
<ul>
  <li>25GbE interfaces</li>
  <li>NVMe controllers</li>
  <li>512GB RAM per node</li>
</ul>

<h2 id="disk-controller">Disk controller</h2>

<p>In a ideal world, we should use a NVMe controller, which QEMU supports, but <a href="https://bugzilla.proxmox.com/show_bug.cgi?id=2255">Proxmox doesn’t support yet</a>. Given that scenario, we’re going with the default SATA controller.</p>

<h2 id="provisioning-of-the-disks">Provisioning of the disks</h2>

<p>For this test, we’ll take 3 nested ESXi instances and add to each 3 x 100GB disks emulating SSD. You can go with more and/or bigger disks, the sky (your wallet?) is the limit.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">STORAGE</span><span class="o">=</span>dstore01
<span class="k">for </span>VM <span class="k">in </span>esxi01 esxi02 esxi03
<span class="k">do</span>
    <span class="c"># We get the ID to operate on the VM</span>
    <span class="nv">VMID</span><span class="o">=</span><span class="si">$(</span>qm list|grep <span class="k">${</span><span class="nv">VM</span><span class="k">}</span>|awk <span class="s1">'{ print $1 }'</span><span class="si">)</span>
    <span class="c"># Adding disks, emulating SSD</span>
    qm <span class="nb">set</span> <span class="k">${</span><span class="nv">VMID</span><span class="k">}</span> <span class="nt">--sata3</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:100,format<span class="o">=</span>qcow2,ssd<span class="o">=</span>1,discard<span class="o">=</span>on <span class="se">\</span>
        <span class="nt">--sata4</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:100,format<span class="o">=</span>qcow2,ssd<span class="o">=</span>1,discard<span class="o">=</span>on <span class="se">\</span>
        <span class="nt">--sata5</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:100,format<span class="o">=</span>qcow2,ssd<span class="o">=</span>1,discard<span class="o">=</span>on
<span class="k">done</span></code></pre></figure>

<h1 id="vsan-setup">vSAN Setup</h1>

<p>You can setup the vSAN Cluster as you would normally do, during vCenter Deployment or after having it up &amp; running.</p>

<p>First warning you would see, is that the NICs are not supported (25GbE is mandated). You can keep moving forward eitherway.</p>

<figure>
  <a href="/assets/img/2023-09-07-nic-speed-warning.png">
  <img src="/assets/img/2023-09-07-nic-speed-warning.png" alt="NIC warning" />
  </a>
  <figcaption><i>Image 1 - NIC warning</i></figcaption>
</figure>

<p>If you keep moving forward, disks will be marked as incompatible. Just claim them manually.</p>

<figure>
  <a href="/assets/img/2023-09-07-incompatible-disk-warning.png">
  <img src="/assets/img/2023-09-07-incompatible-disk-warning.png" alt="Disk marked as not compatible" />
  </a>
  <figcaption><i>Image 2 - Disk marked as not compatible</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-09-07-disk-claiming.png">
  <img src="/assets/img/2023-09-07-disk-claiming.png" alt="Manual claim" />
  </a>
  <figcaption><i>Image 3 - Manual claim</i></figcaption>
</figure>

<h1 id="outcome">Outcome</h1>

<p>After some minutes, you should have your working-but-not-supported ESXi-nested-on-PVE vSAN cluster.</p>

<p>Have fun!</p>]]></content><author><name>Ciro Iriarte</name></author><category term="HomeLab" /><category term="ESXi" /><category term="Proxmox" /><category term="Nested" /><category term="Virtualization" /><category term="vSAN" /><summary type="html"><![CDATA[Nested vSAN ESA cluster for testing purposes]]></summary></entry><entry><title type="html">ESXi on Proxmox as nested hypervisor</title><link href="https://iriarte.it/homelab/2023/09/05/esxi-on-proxmox-as-nested-hypervisor.html" rel="alternate" type="text/html" title="ESXi on Proxmox as nested hypervisor" /><published>2023-09-05T00:00:00-04:00</published><updated>2023-09-05T00:00:00-04:00</updated><id>https://iriarte.it/homelab/2023/09/05/esxi-on-proxmox-as-nested-hypervisor</id><content type="html" xml:base="https://iriarte.it/homelab/2023/09/05/esxi-on-proxmox-as-nested-hypervisor.html"><![CDATA[<p>Thanks to the work of <a href="https://williamlam.com/nested-virtualization">William Lam</a>, running ESXi on top of ESXi for testing/lab purposes is really straightforward.</p>

<p>Now, if for some reason you cannot install ESXi on your home lab machine, and like me you use Proxmox, this post is for you.</p>

<p><a href="https://www.proxmox.com/">Proxmox Virtual Environment</a> or PVE is a nice KVM-based virtualization solution from <a href="https://www.proxmox.com/en/about/company">Proxmox Server Solutions GmbH</a>. It’s free to use, and has a wide userbase.</p>

<p>The starting point would be a properly setup <a href="https://pve.proxmox.com/wiki/Nested_Virtualization">Proxmox host for nested virtualization</a>. After you get it up &amp; running, we’re ready to move forward.</p>

<h1 id="the-esxi-appliance">The ESXi appliance</h1>
<p>This <a href="https://williamlam.com/nested-virtualization/nested-esxi-virtual-appliance">ESXi appliance</a> is pre-configured to simplify deployment as a nested hypervisor. With each new vSphere release, a matching appliance is created and published.</p>

<h1 id="downloading-the-appliance">Downloading the appliance</h1>
<p>The mentioned appliances are published via a public vSphere Content Library. To get a list of the available releases, we can run the following command from your PVE host:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">BASEURL</span><span class="o">=</span>https://download3.vmware.com/software/vmw-tools
curl <span class="nt">-s</span> <span class="k">${</span><span class="nv">BASEURL</span><span class="k">}</span>/items.json |grep <span class="nt">-A</span> 1 <span class="s2">"hrefs"</span>|grep <span class="nt">-vE</span> <span class="s2">"</span><span class="se">\-\-</span><span class="s2">|hrefs"</span>|sort |cut <span class="nt">-f</span> 2 <span class="nt">-d</span> <span class="s1">'"'</span>|cut <span class="nt">-f</span> 1 <span class="nt">-d</span> <span class="s2">"/"</span>|sort <span class="nt">-u</span></code></pre></figure>

<p>The output should be similar to:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">Nested_ESXi6.0u3_Appliance_Template_v1.0
Nested_ESXi6.5d_Appliance_Template_v1.0
Nested_ESXi6.5u1_Appliance_Template_v1.0
Nested_ESXi6.5u2_Appliance_Template_v1.0
Nested_ESXi6.5u3_Appliance_Template_v1.0
Nested_ESXi6.7_Appliance_Template_v1.0
Nested_ESXi6.7u1_Appliance_Template_v1.0
Nested_ESXi6.7u2_Appliance_Template_v1.0
Nested_ESXi6.7u3_Appliance_Template_v1.0
Nested_ESXi7.0_Appliance_Template_v1.0
Nested_ESXi7.0u1_Appliance_Template_v1.0
Nested_ESXi7.0u1d_Appliance_Template_v1.0
Nested_ESXi7.0u2a_Appliance_Template_v2.0
Nested_ESXi7.0u2_Appliance_Template_v1.0
Nested_ESXi7.0u3_Appliance_Template_v1.0
Nested_ESXi7.0u3c_Appliance_Template_v1.0
Nested_ESXi7.0u3d_Appliance_Template_v1.0
Nested_ESXi7.0u3e_Appliance_Template_v1.0
Nested_ESXi7.0u3f_Appliance_Template_v1.0
Nested_ESXi7.0u3g_Appliance_Template_v1.0
Nested_ESXi7.0u3i_Appliance_Template_v1.0
Nested_ESXi7.0u3j_Appliance_Template_v1.0
Nested_ESXi7.0u3k_Appliance_Template_v1.0
Nested_ESXi7.0u3l_Appliance_Template_v1.0
Nested_ESXi7.0u3m_Appliance_Template_v1.0
Nested_ESXi8.0a_Appliance_Template_v1.0
Nested_ESXi8.0b_Appliance_Template_v1.0
Nested_ESXi8.0c_Appliance_Template_v1.0
Nested_ESXi8.0_IA_Appliance_Template_v2.0
Nested_ESXi8.0u1a_Appliance_Template_v1
Nested_ESXi8.0u1_Appliance_Template_v1.0</code></pre></figure>

<p>Pick the one you would like to use, and download all the related files:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c"># We create the destination directory</span>
<span class="nb">mkdir </span>esxi<span class="p">;</span><span class="nb">cd </span>esxi
<span class="c"># Variable holding the chosen appliance name</span>
<span class="nv">APPLIANCE</span><span class="o">=</span>Nested_ESXi8.0u1a_Appliance_Template_v1
<span class="c"># Get the list of the files we'll fetch</span>
<span class="nv">FILES</span><span class="o">=</span><span class="si">$(</span>curl <span class="nt">-s</span> <span class="k">${</span><span class="nv">BASEURL</span><span class="k">}</span>/items.json |grep <span class="nt">-A</span> 1 <span class="s2">"hrefs"</span>|grep <span class="nt">-vE</span> <span class="s2">"</span><span class="se">\-\-</span><span class="s2">|hrefs"</span>|sort |cut <span class="nt">-f</span> 2 <span class="nt">-d</span> <span class="s1">'"'</span>|grep <span class="nv">$APPLIANCE</span><span class="si">)</span>
<span class="c"># For loop to download file by file</span>
<span class="k">for </span>F <span class="k">in</span> <span class="nv">$FILES</span>
<span class="k">do
    </span>wget <span class="k">${</span><span class="nv">BASEURL</span><span class="k">}</span>/<span class="k">${</span><span class="nv">F</span><span class="k">}</span>
<span class="k">done</span></code></pre></figure>

<p>You should end up with a group of files similar to:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@bigiron:~/esxi# <span class="nb">ls</span> <span class="nt">-l</span>
total 1696791
<span class="nt">-rw-r--r--</span> 1 root root   865786368 Jun  2 11:38 Nested_ESXi8.0u1a_Appliance_Template_v1-disk1.vmdk
<span class="nt">-rw-r--r--</span> 1 root root       68096 Jun  2 11:35 Nested_ESXi8.0u1a_Appliance_Template_v1-disk2.vmdk
<span class="nt">-rw-r--r--</span> 1 root root       68608 Jun  2 11:35 Nested_ESXi8.0u1a_Appliance_Template_v1-disk3.vmdk
<span class="nt">-rw-r--r--</span> 1 root root         389 Jun  2 11:35 Nested_ESXi8.0u1a_Appliance_Template_v1.mf
<span class="nt">-rw-r--r--</span> 1 root root       54052 Jun  2 11:35 Nested_ESXi8.0u1a_Appliance_Template_v1.ovf</code></pre></figure>

<h1 id="importing-the-appliance">Importing the appliance</h1>

<p>After downloading the files, we’ll create a virtual machine using the OVF as source. It’s mostly used to import the disk images, large parts of the file are actually ignored (advanced configuration parameters for example are not taken into account). For such task &amp; initial VM configuration, I’ve created a quick script (create-esxi.sh) to be run in your PVE host.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="c">#!/bin/bash</span>

<span class="c"># We need at least 1 VLAN for management &amp; another for vSAN traffic</span>
<span class="nv">VLANLAB</span><span class="o">=</span>302
<span class="nv">VLANVSAN</span><span class="o">=</span>303
<span class="c"># Bridge interface to use (VLAN aware)</span>
<span class="nv">BRIDGE</span><span class="o">=</span>vmbr0
<span class="c"># Storage destination for the appliance to live in</span>
<span class="nv">STORAGE</span><span class="o">=</span>dstore01
<span class="c"># Source OVF file, descriptor of the appliance</span>
<span class="nv">OVF</span><span class="o">=</span>esxi/Nested_ESXi8.0u1a_Appliance_Template_v1.ovf
<span class="c"># Template/appliance name as defined in the OVF file, for us to later rename the VM.</span>
<span class="nv">TMPL_NAME</span><span class="o">=</span><span class="si">$(</span><span class="nb">grep</span> <span class="s2">"&lt;Name&gt;"</span> <span class="nv">$OVF</span>| <span class="nb">cut</span> <span class="nt">-f</span> 2 <span class="nt">-d</span> <span class="s2">"&gt;"</span> |cut <span class="nt">-f</span> 1 <span class="nt">-d</span> <span class="s2">"&lt;"</span> |sed <span class="s1">'s/_//g'</span><span class="si">)</span>

<span class="c"># We need at least the name of the VM to create as script parameter</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-ne</span> 1 <span class="o">]</span>
<span class="k">then
        </span><span class="nb">echo</span> <span class="s2">"usage </span><span class="nv">$0</span><span class="s2"> &lt;vm name&gt;"</span>
        <span class="nb">exit </span>1
<span class="k">else
        </span><span class="nv">VMNAME</span><span class="o">=</span><span class="nv">$1</span>
<span class="k">fi</span>

<span class="c"># Basic VM creation and disk files import</span>
qm importovf <span class="si">$(</span>pvesh get /cluster/nextid<span class="si">)</span> <span class="k">${</span><span class="nv">OVF</span><span class="k">}</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span> <span class="nt">-format</span> qcow2

<span class="c"># We get the ID of the created VM</span>
<span class="nv">NEWVM</span><span class="o">=</span><span class="si">$(</span>qm list|grep <span class="k">${</span><span class="nv">TMPL_NAME</span><span class="k">}</span>|awk <span class="s1">'{ print $1 }'</span><span class="si">)</span>

<span class="c"># We define initial configuration. In this case, 12 cores in total, 2 sockets &amp; 64GB of RAM. EFI boot.</span>
qm <span class="nb">set</span> <span class="nv">$NEWVM</span> <span class="nt">--name</span> <span class="nv">$VMNAME</span> <span class="nt">--bios</span> ovmf <span class="nt">--machine</span> q35 <span class="se">\</span>
<span class="nt">--numa</span> 1 <span class="nt">--sockets</span> 2 <span class="nt">--cores</span> 6 <span class="nt">--cpu</span> <span class="nv">cputype</span><span class="o">=</span>host <span class="se">\</span>
<span class="nt">--scsihw</span> pvscsi <span class="se">\</span>
<span class="nt">--memory</span> 65536 <span class="nt">--hugepages</span> 1024 <span class="se">\</span>
<span class="nt">--efidisk0</span> <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>:0,efitype<span class="o">=</span>4m,format<span class="o">=</span>qcow2

<span class="c"># We add two network interfaces with the correct VLAN mapping</span>
qm <span class="nb">set</span> <span class="nv">$NEWVM</span> <span class="se">\</span>
<span class="nt">--net0</span> <span class="nv">model</span><span class="o">=</span>vmxnet3,bridge<span class="o">=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span>,firewall<span class="o">=</span>0,tag<span class="o">=</span><span class="k">${</span><span class="nv">VLANLAB</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--net1</span> <span class="nv">model</span><span class="o">=</span>vmxnet3,bridge<span class="o">=</span><span class="k">${</span><span class="nv">BRIDGE</span><span class="k">}</span>,firewall<span class="o">=</span>0,tag<span class="o">=</span><span class="k">${</span><span class="nv">VLANVSAN</span><span class="k">}</span>

<span class="c"># The import command attaches the disks to a SCSI controller, the VM doesn't recognize the VMWare PVSCSI controller unluckily. We detach disks from that controller.</span>
<span class="k">for </span>DISK <span class="k">in</span> <span class="si">$(</span>qm config <span class="nv">$NEWVM</span> |grep ^scsi|grep <span class="nv">$NEWVM</span>|cut <span class="nt">-f</span> 1 <span class="nt">-d</span> <span class="s2">":"</span><span class="si">)</span>
<span class="k">do
        </span>qm <span class="nb">set</span> <span class="nv">$NEWVM</span> <span class="nt">--delete</span> <span class="nv">$DISK</span>
<span class="k">done</span>

<span class="c"># Reattach of disks to SATA controller. The appliance is happier with that one.</span>
<span class="nv">n</span><span class="o">=</span>0
<span class="k">for </span>DISK <span class="k">in</span> <span class="si">$(</span>qm config <span class="nv">$NEWVM</span>|grep ^unused|awk <span class="s1">'{ print $2 }'</span><span class="si">)</span>
<span class="k">do
        </span>qm <span class="nb">set</span> <span class="nv">$NEWVM</span> <span class="nt">-sata</span><span class="k">${</span><span class="nv">n</span><span class="k">}</span> <span class="nv">$DISK</span>
        <span class="nv">n</span><span class="o">=</span><span class="k">$((</span> <span class="nv">$n</span> <span class="o">+</span> <span class="m">1</span> <span class="k">))</span>
<span class="k">done</span>

<span class="c"># Define boot disk</span>
qm <span class="nb">set</span> <span class="nv">$NEWVM</span> <span class="nt">--boot</span> <span class="nv">order</span><span class="o">=</span>sata0

<span class="c"># Start VM</span>
qm start <span class="nv">$NEWVM</span></code></pre></figure>

<p>To create 3 virtual machines for our initial cluster, we just execute:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">./create-esxi.sh esxi01
./create-esxi.sh esxi02
./create-esxi.sh esxi03</code></pre></figure>

<p>After that, manual initial setup of ESXi is required (hostname, static IP or sending hostname in DHCP request, etc)</p>

<h1 id="outcome">Outcome</h1>

<p>In the end, we should have the 3 VMs up &amp; running and ready for testing.</p>

<figure>
  <a href="/assets/img/2023-09-05-nested-esxi-vm-list.png">
  <img src="/assets/img/2023-09-05-nested-esxi-vm-list.png" alt="PVE web console" />
  </a>
  <figcaption><i>Image 1 - Created Virtual Machines</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-09-05-nested-esxi-console.png">
  <img src="/assets/img/2023-09-05-nested-esxi-console.png" alt="VM console" />
  </a>
  <figcaption><i>Image 2 - VM console</i></figcaption>
</figure>

<figure>
  <a href="/assets/img/2023-09-05-nested-esxi-webUI.png">
  <img src="/assets/img/2023-09-05-nested-esxi-webUI.png" alt="ESXi Web UI" />
  </a>
  <figcaption><i>Image 3 - ESXi web UI</i></figcaption>
</figure>

<h1 id="todo-list">ToDo list</h1>

<h3 id="1-advanced-configuration-parameters">1. Advanced configuration parameters</h3>
<p>Part of the “ready for nested virtualization” experience involves a series of advanced configurations that come defined in the OVF file. Some of them are only relevant to ESXi hosts for the appliance, but some might need to be “translated” to QEMU/KVM configuration adjustments.</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">root@bigiron:~/esxi# <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"vmw:ExtraConfig|vmw:Config"</span> Nested_ESXi8.0u1a_Appliance_Template_v1.ovf
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"slotInfo.pciSlotNumber"</span> vmw:value<span class="o">=</span><span class="s2">"16"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"useAutoDetect"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"videoRamSizeInKB"</span> vmw:value<span class="o">=</span><span class="s2">"4096"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"enable3DSupport"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"use3dRenderer"</span> vmw:value<span class="o">=</span><span class="s2">"automatic"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"graphicsMemorySizeInKB"</span> vmw:value<span class="o">=</span><span class="s2">"262144"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"slotInfo.pciSlotNumber"</span> vmw:value<span class="o">=</span><span class="s2">"35"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"allowUnrestrictedCommunication"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"backing.exclusive"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"connectable.allowGuestControl"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"backing.writeThrough"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"backing.writeThrough"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"backing.writeThrough"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"slotInfo.pciSlotNumber"</span> vmw:value<span class="o">=</span><span class="s2">"33"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"wakeOnLanEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"connectable.allowGuestControl"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"uptCompatibilityEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"uptv2Enabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"wakeOnLanEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"connectable.allowGuestControl"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"uptCompatibilityEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
        &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"uptv2Enabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"cpuHotAddEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"cpuHotRemoveEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"memoryHotAddEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"firmware"</span> vmw:value<span class="o">=</span><span class="s2">"efi"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"cpuAllocation.shares.shares"</span> vmw:value<span class="o">=</span><span class="s2">"2000"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"cpuAllocation.shares.level"</span> vmw:value<span class="o">=</span><span class="s2">"normal"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"simultaneousThreads"</span> vmw:value<span class="o">=</span><span class="s2">"1"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.syncTimeWithHost"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.syncTimeWithHostAllowed"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.afterPowerOn"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.afterResume"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.beforeGuestShutdown"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.beforeGuestStandby"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"tools.toolsUpgradePolicy"</span> vmw:value<span class="o">=</span><span class="s2">"upgradeAtPowerCycle"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"fixedPassthruHotPlugEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"powerOpInfo.powerOffType"</span> vmw:value<span class="o">=</span><span class="s2">"soft"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"powerOpInfo.resetType"</span> vmw:value<span class="o">=</span><span class="s2">"soft"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"powerOpInfo.suspendType"</span> vmw:value<span class="o">=</span><span class="s2">"soft"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"nestedHVEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"vPMCEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"virtualICH7MPresent"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"virtualSMCPresent"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"flags.vvtdEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"flags.vbsEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"bootOptions.efiSecureBootEnabled"</span> vmw:value<span class="o">=</span><span class="s2">"false"</span>/&gt;
      &lt;vmw:Config ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"powerOpInfo.standbyAction"</span> vmw:value<span class="o">=</span><span class="s2">"checkpoint"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"disk.enableuuid"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ehci.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"34"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.bsdname"</span> vmw:value<span class="o">=</span><span class="s2">"en0"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.connectiontype"</span> vmw:value<span class="o">=</span><span class="s2">"nat"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.displayname"</span> vmw:value<span class="o">=</span><span class="s2">"Ethernet"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.filter4.name"</span> vmw:value<span class="o">=</span><span class="s2">"dvfilter-maclearn"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.filter4.onfailure"</span> vmw:value<span class="o">=</span><span class="s2">"failOpen"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.linkstatepropagation.enable"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet0.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"33"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet1.filter4.name"</span> vmw:value<span class="o">=</span><span class="s2">"dvfilter-maclearn"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"ethernet1.filter4.onfailure"</span> vmw:value<span class="o">=</span><span class="s2">"failOpen"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"gui.fullscreenatpoweron"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"gui.viewmodeatpoweron"</span> vmw:value<span class="o">=</span><span class="s2">"windowed"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"hgfs.linkrootshare"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"hgfs.maprootshare"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"hpet0.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"isolation.tools.hgfs.disable"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"msg.autoanswer"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"parallel0.autodetect"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge0.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"17"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge0.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge4.functions"</span> vmw:value<span class="o">=</span><span class="s2">"8"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge4.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"21"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge4.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge4.virtualdev"</span> vmw:value<span class="o">=</span><span class="s2">"pcieRootPort"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge5.functions"</span> vmw:value<span class="o">=</span><span class="s2">"8"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge5.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"22"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge5.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge5.virtualdev"</span> vmw:value<span class="o">=</span><span class="s2">"pcieRootPort"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge6.functions"</span> vmw:value<span class="o">=</span><span class="s2">"8"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge6.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"23"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge6.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge6.virtualdev"</span> vmw:value<span class="o">=</span><span class="s2">"pcieRootPort"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge7.functions"</span> vmw:value<span class="o">=</span><span class="s2">"8"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge7.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"24"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge7.present"</span> vmw:value<span class="o">=</span><span class="s2">"TRUE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"pcibridge7.virtualdev"</span> vmw:value<span class="o">=</span><span class="s2">"pcieRootPort"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"proxyapps.publishtohost"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"remotedisplay.vnc.enabled"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"replay.supported"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"scsi0.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"16"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"scsi0:1.virtualSSD"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"scsi0:2.virtualSSD"</span> vmw:value<span class="o">=</span><span class="s2">"true"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"serial0.autodetect"</span> vmw:value<span class="o">=</span><span class="s2">"FALSE"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"usb.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"32"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"uuid.action"</span> vmw:value<span class="o">=</span><span class="s2">"create"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"virtualhw.productcompatibility"</span> vmw:value<span class="o">=</span><span class="s2">"hosted"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"vmci0.pcislotnumber"</span> vmw:value<span class="o">=</span><span class="s2">"35"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"vmotion.checkpointfbsize"</span> vmw:value<span class="o">=</span><span class="s2">"65536000"</span>/&gt;
      &lt;vmw:ExtraConfig ovf:required<span class="o">=</span><span class="s2">"false"</span> vmw:key<span class="o">=</span><span class="s2">"migrate.hostLog"</span> vmw:value<span class="o">=</span><span class="s2">"./Nested_ESXi8.0u1a_Appliance_Template-222fd488.hlog"</span>/&gt;</code></pre></figure>

<p>With more time, I’ll take a look at the list and provide an update if needed.</p>

<h3 id="2-cloud-init">2. cloud-init</h3>

<p>PVE provides support for cloud-init as a way to automate VM provisioning &amp; initial configuration. ESXi doesn’t provide a cloud-init agent out of the box.</p>

<p>I’ve found a basic implementation of a cloud-init agent for ESXi by <a href="https://github.com/goneri">Gonéri Le Bouder</a>, which some people have reported to be working to deploy ESXi on top of Openstack. Pending to review how to inject that script in our freshly downloaded appliance (encrypted local.tgz proven to be cumbersome for this usecase).</p>

<ul>
  <li><a href="https://github.com/goneri/esxi-cloud-init">https://github.com/goneri/esxi-cloud-init</a></li>
  <li><a href="https://github.com/virt-lightning/esxi-cloud-images">https://github.com/virt-lightning/esxi-cloud-images</a></li>
</ul>]]></content><author><name>Ciro Iriarte</name></author><category term="HomeLab" /><category term="ESXi" /><category term="Proxmox" /><category term="Nested" /><category term="Virtualization" /><summary type="html"><![CDATA[Step by step implementation of a nested ESXi machine running on Proxmox/PVE KVM host.]]></summary></entry></feed>