Mastering Proxmox VE

More in this magazine

• Community Experience DistilledMastering Proxmox Master Proxmox VE to effectively implement server virtualization technology within your network
• Mastering ProxmoxMaster Proxmox VE to effectively implement server virtualization technology within your network Wasim Ahmed
• Mastering ProxmoxCopyright © 2014 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
• CreditsProject Coordinator Neha Bhatnagar Author
• About the AuthorWasim Ahmed, born in Bangladesh and now a citizen of Canada, is a veteran of the IT world. He was introduced to computers in the year 1992 and never looked back. Wasim has deep knowledge and understanding of network virtualization, big data storage, and network security. By profession, Wasim is the CEO of an IT support and cloud service provider company based in Calgary, Alberta. He serves many companies and organizations through his company on a daily basis. Wasim's strength comes from the experience he gained through learning and serving continually. Wasim strives on finding the most effective solution at the most competitive price point. He hand-built over a dozen enterprise production virtual infrastructures using Proxmox and Ceph storage system. Wasim is notoriously known not to simply accept a technology based on its description alone, but put them through rigorous tests to check their validity. Any new technology that his company provides goes through months of continuous testing before it is accepted. Proxmox made the cut superbly.
• About the ReviewersRocco Alfonzetti Jr. is an IT consultant for small businesses and has specialized in Linux and open source solutions for the last 15 years. Currently, he works for a software development company as an e-mail security expert. He lives in rural Connecticut with his wife and three children, and in his spare time, he enjoys beekeeping, raising chickens, and gardening. Alessio Bravi has been playing with bits since he was five. He started programming at the age of six and soon focused his attention towards network administration and IT systems security in the best growing-up period of the Internet.
• Page 7Oleg Butovich is a freelance senior software developer with a passion for virtualization technologies. He has over 15 years of experience in the industry. He has worked on booking systems, trading platforms, laser image generators, digital media systems, medical and life science imaging systems, automatic inspection systems, and embedded systems. Daniel Lench is a self-proclaimed "fixer of all things". He is drawn to challenges, both physical and theoretical. His background includes acting as an artisan at a state museum, a production manager at a high-volume cabinet shop, AutoCAD expert for civil engineering firms and government agencies, and almost two decades of being professionally involved in the IT industry. In 2008, the challenge was to keep files in sync between multiple computers in real time. Since then he has been focused on finding the best answer. The NoFolder Project is an open source, real-time, private cloud-based backup, file synchronization, and collaboration service that is self hosted and administered in small business and enterprise settings. NoFolder addressed the policy and privacy concerns over using third-party services to store and share data. The project is for those concerned about data, the collaboration with it, and the preservation of it. The company maintains offices in the U.S. and the U.K. with additional resources in Sweden, Austria, and South Africa. Daniel is the founder as well as the CEO for NoFolder Ltd.
• www.PacktPub.comSupport files, eBooks, discount offers, and more You might want to visit www.Packt Pub.com for support files and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.Packt Pub. com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
• Page 9www.allitebooks.com
• Page 10To my dear wife, Brigitta, whose love, constant support, and unshakeable faith in me has allowed me to do what I do best, while she took care of the rest. www.allitebooks.com
• Page 11
• PrefaceChapter 1: Dive into the Virtual World with Proxmox Proxmox cluster required
• Table of ContentsProxmox subscription Attaching shared storage Adding virtual machines Main virtual machine Creating a KVM virtual machine Creating an OpenVZ virtual machine Proxmox cloning/template
• Page 14The raw image type The .vmdk image type Image file manipulation Resizing virtual disk image Moving a virtual disk image Storage types in Proxmox
• Table of ContentsNetworking components in Proxmox Virtual Network Interface Card (VNIC)
• Page 16Reasons to use Ceph Virtual Ceph for training The Ceph components Physical node Maps Cluster map
• Table of ContentsLearning Ceph's CRUSH map Extracting the CRUSH map Decompiling the CRUSH map Editing the CRUSH map Compiling the CRUSH map
• Chapter 9: Proxmox TroubleshootingMain cluster issues GUI shows everything is offline
• Table of ContentsKVM virtual machine issues 245
• Page 20Index Chapter 10: Putting It All Together
• Page 21
• PrefaceThis book is well overdue in the world of virtualization. When I first came in contact with Proxmox several years ago, I did not have anything to fall back on other than Proxmox Wiki and forum. I learned Proxmox through lots of trial and error and very much had to reinvent wheels on my own in some cases. Since a lot of us went through the frustration and I personally do not feel others should have to invest a lot of time just to get to know Proxmox the hard way, this book has been written. This book shows the inner workings of Proxmox including virtual network components, shared storage systems, nested virtualization, complex network topologies, and so on. With this book, we hope that the reader will be able to better equip themselves to face any virtualization challenges of any virtual infrastructure.
• PrefaceChapter 7, High Availability Storage for High Availability Cluster, explains a step-by-step process of setting up the Ceph cluster to be used as a shared storage system. Chapter 8, Proxmox Production Level Setup, explains the type of hardware that should be and can be used in a production level Proxmox cluster setup.
• <clusternode name="pmxvm02" votes="1" nodeid="2"/> </clusternodes></cluster> Any command-line input or output is written as follows:
• PrefaceErrata Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub. com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
• Dive into the Virtual World with Proxmox1 With the rise in energy costs and the advancement of computer processing power and memory capacity, running separate, under-utilized server hardware with specific roles is no longer a luxury, and thus was born the technology we so affectionately call virtualization. The term virtualization mainly refers to virtualizing servers, but of late it is also being used for network or infrastructure virtualization. Server and network virtualization enabled us to truly create a virtual world where anything is possible. Virtual data centers, virtual storage systems, and virtual desktop PCs are just a few real-world applications where virtualization is being heavily used. Although nowadays the world is being swept by virtualization, the term virtualization itself is not new. The concept was used in mainframes of the 1960s, where virtualization was a way to logically divide the mainframe's resources for different application processing.
• Dive into the Virtual World with ProxmoxThe Proxmox hypervisor is one of the best kept secrets in the modern computer world. The reason is simple. It allows for the building of an enterprise business-class virtual infrastructure at a small business-class price tag without sacrificing stability, performance, and ease of use. Whether it is a massive data center to serve millions of people, a small educational institution, or a home serving important family members, Proxmox can fulfil the needs of just about any situation. Even a novice networker can get a stable virtualization platform up and running in less than an hour. A Proxmox cluster consists of two or more computer nodes with Proxmox as the operating system and connected in the same network. A virtual machine can migrate from one node to another in the same cluster, which allows redundancy should a node fail for any reason. Refer to the following diagram of a very basic two-node Proxmox cluster with FreeNAS shared storage. Please note that while this form of setup is good enough for learning purposes, it may not be enough for a production environment where uptime is essential. In later chapters, we will see how to add more Proxmox nodes and storage nodes into the cluster to ensure redundancy.
• Chapter 1Proxmox cluster required A hands-on approach has been followed throughout this book to allow the reader to learn Proxmox in a practical way. If you do not have a Proxmox cluster set up or no access to an existing cluster, you can set up a basic-level Proxmox cluster by following the installation instructions laid out in the Setting up a basic cluster section of this chapter. If you already have a cluster, follow along from the next section, The Proxmox Graphical User Interface (GUI).
• Dive into the Virtual World with ProxmoxThe various fields marked in the previous screenshot are as follows: 1 shows the URL to access the Proxmox GUI through a browser
• Chapter 1Some features of the Proxmox GUI, such as VNC console and shell, require Java or Iced Tea (http://icedtea.classpath.org/wiki/Main_Page) to be installed on the computer you are accessing the GUI from. The GUI works great with Firefox and Google Chrome. The latest version of Internet Explorer may have an issue with functioning properly if not in compatibility mode. The Proxmox GUI also works with the Opera browser. Menu chart
• Dive into the Virtual World with ProxmoxThe Datacenter menu In the Proxmox GUI, Datacenter is the main level folder of Proxmox nodes'/VMs' tree. Each Datacenter folder can only hold one Proxmox cluster.
• DatacenterStorage Backup Search
• Dive into the Virtual World with ProxmoxThe Backup tab Although storage can be changed at a later time, it is very important to have a clear understanding of all storage types for better planning in the beginning. All storage types have both advantages and disadvantages. The ultimate decision will come down to performance, stability, and redundancy.
• Chapter 1Create: Backup Job Node:
• Dive into the Virtual World with ProxmoxNode-specific tabs The Summary tab
• The Network tabChapter 1 To use the Network tab, navigate to GUI | Server View | Node | Network.
• Dive into the Virtual World with ProxmoxThe following screenshot shows the Syslog option: Server View
• The Subscription tabChapter 1 To use the Subscription tab, navigate to GUI | Server View | Node | Subscription. Proxmox can be downloaded and used for free without any restriction for any feature. It is by no means a trialware, shareware, or an n-day evaluation hypervisor. However, Proxmox also has a subscription model, which allows enterprise-class repositories. The free version of Proxmox only comes with standard repositories. The main difference between enterprise and standard repositories is that enterprise repositories go through a higher level of testing to ensure a very stable cluster environment. The following screenshot shows the Subscription tab:
• Dive into the Virtual World with ProxmoxThe Updates tab To use the Updates tab, navigate to GUI | Server View | Node | Updates.
• Chapter 1Starting with Proxmox VE 3.2, Ceph server is added as technology preview. This allows both Proxmox and Ceph to co-exist on the same node. Ceph itself does not come with any graphical user interface to manage Ceph storage, with the exception being the subscription version of Ceph. Proxmox enables us to manage Ceph cluster almost entirely from the Proxmox GUI. Currently CrushMAP cannot be edited and multiple Ceph clusters cannot be managed through the GUI. The following screenshot shows the Ceph menu tab along with Ceph-related tabs: Network DNS Time Syslog Bootlog Task History UBC Subscription Updates Ceph HEALTH WARN 137 pcs degraded;
• Dive into the Virtual World with ProxmoxThe Summary tab Virtual machine tabs
• Chapter 1Virtual Machine 101 ('pmxMS01') on node 'pmxvm02' Start Shutdown Stop Reset Hardware
• Dive into the Virtual World with ProxmoxThe Options tab Move Disk is the safest way to move a virtual hard drive from one storage to another. If the virtual disk is on a shared storage, then live migration of the virtual disk is possible, allowing a great amount of time saving.
• The Backup tabChapter 1 A good backup plan is the first line of defense against any disaster, which can cause major or minor data loss. In our ultra-modern digital world, data is much more valuable than ever before. Every virtual environment administrator struggles with backup strategy of his/her virtual environment. The following screenshot shows the Backup tab:
• Dive into the Virtual World with ProxmoxThe Snapshots tab Proxmox Snapshots is a way to roll back a KVM-based virtual machine to a previous state. Although it provides similar protection to Proxmox Backup, it comes with speed. Proxmox Snapshot is extremely fast when compared with Proxmox Backup, thus allowing a user to take several snapshots a day. The following screenshot shows the Snapshots tab:
• Chapter 1A common scenario of permission usage is in an office setup where there is one accounting virtual machine and multiple staff need to access data. A permission can be set either at the user or the group level. Setting up a basic cluster
• Dive into the Virtual World with ProxmoxThe hardware list The following is a list of hardware components that we will need to put together our first basic Proxmox cluster. If you already have some components you would like to use to set up your cluster, it is important to check if they will support virtualization. Not all hardware platforms support virtualization, especially if they are quite old. To get details on how to check your components, visit http://virt-tools.org/ learning/check-hardware-virt/.
• Hardware setupChapter 1 The next diagram is a network diagram of a basic Proxmox cluster. We will start with two node clusters with one shared storage setup with either Ubuntu or FreeNAS. The setup in the illustration is a guideline only. Depending on the level of experience, budget, and available hardware on hand, you can set up any way you see fit. Regardless of whatever setup you use, it should meet the following requirements:
• Dive into the Virtual World with Proxmox3. Boot the node from the Proxmox installation disc. 4. Follow along the Proxmox graphical installation process. Enter the IP address 192.168.145.1, or any other subnet you wish, when prompted. Also enter pmxvm1.domain.com or any other hostname that you choose to use.
• Chapter 1Restarting pve cluster filesystem: pve-cluster [dcdb] notice: wrote new cluster config '/etc/cluster/cluster.conf' Starting cluster:
• Dive into the Virtual World with ProxmoxNode name: pxvm01 Node ID: 1
• Proxmox subscriptionChapter 1 On a clean installed Proxmox node, a paid subscription-based repository is enabled by default. When you log in to the Proxmox GUI, the following message will pop up on entering login information:
• Dive into the Virtual World with ProxmoxAfter you have the choice of shared storage server setup, attach the storage with Proxmox by navigating to Datacenter | Storage. There should be three shares as shown in the following table: Share type
• Main virtual machineChapter 1 The virtual machine we are going to create will act as main server for the rest of the virtual machines in the cluster. This will provide services such as DHCP, DNS, and so on. You can use any Linux flavor you are familiar with to get the DHCP/ DNS Server set up. The ClearOS Community edition is a great choice since it allows putting all services in one machine and actually works very well.
• Dive into the Virtual World with Proxmox4. Click on the Select File... button to select the ISO image, and then click on the Upload button. After uploading, the ISO will show up on the content page as shown in the following screenshot: Summary Content Permissions
• Creating a KVM virtual machineChapter 1 After the main server is set up, we are now going to create a second virtual machine with Ubuntu as the operating system. Proxmox has a cloning feature, which saves lot of time when deploying VMs with the same operating system and configuration. We will use the Ubuntu virtual machine as the template for all Linux-based VMs throughout this book. Create the Ubuntu VM with the following configuration:
• Dive into the Virtual World with ProxmoxGo to http://openvz.org/Main_Page for more details on OpenVZ. Unlike a KVM virtual machine, OpenVZ containers cannot be installed using an ISO image. Proxmox uses templates to create OpenVZ container virtual machines and comes with the very nice feature of templates repository. At the time of this writing, the repository has close to 400 templates ready to download through the Proxmox GUI.
• Chapter 1Templates Package
• Dive into the Virtual World with ProxmoxOpenVZ containers cannot be cloned for mass deployment. If such mass deployment is required, then the container can be backed up and restored with different VM IDs as many times as required. With all three of the virtual machines created, the Proxmox cluster GUI should look like the following screenshot:
• Page 60A template has a distinct icon as seen in the following screenshot, which easily identifies it from a standard virtual machine. Just create a VM with desired configuration, and then by the touch of a mouse click, turn the VM into a template. Whenever a new VM is required, just clone the template. epmxvm01
• Dive into the Virtual World with Proxmox2. Click on Convert to template as shown in the following screenshot. This will convert the VM into a template that can be used to clone an unlimited number of virtual machines. While creating a template, keep in mind that a template itself cannot be used as a virtual machine. But it can be migrated to different hosts just like a virtual machine. Datacenter
• Chapter 1The most important option to notice in this menu is the Mode option. A clone can be created from a template using either Full Clone or Linked Clone. Full Clone versus Linked Clone
• Dive into the Virtual World with ProxmoxWith the migration option, a running VM can be moved to another node without a single downtime. During live migration, VM does not experience any major slowdown. After the node reboots, simply migrate the VMs back to the original node. Any offline VMs can also be moved with ease. Migrate VM 101
• Chapter 1With the introductory chapter out of the way, in the next chapter, we will take a look at what is under the hood of Proxmox hypervisor. We will see how the Proxmox folder structure is laid out to hold some of the important files, which make Proxmox run so effectively. Most importantly, we will go deeper into some of the configuration and see their function line by line. In order to build a complex enterprise-class Proxmox cluster, it is important to be quite familiar with these configurations. Proxmox cluster can be tweaked and tailored further beyond the GUI through these files. For more information, you can visit http://www.masteringproxmox.com/. You can use this forum for discussing about Proxmox and related topics.
• Page 65
• Proxmox Under the Hood2 In the previous chapter, we have seen how a Proxmox GUI looks like and set up a basic cluster, which we are going to use for hands-on learning of Proxmox. In this chapter, we will look at how configuration files hold a Proxmox virtualization platform together, files to be used for advanced configuration, and how to troubleshoot a Proxmox platform. Proxmox is built on Debian Linux, which is very stable with a large active community. So, it inherited the heavy dependency on configuration or the .conf files as we Linux users lovingly like to call them. The Proxmox GUI provides you with the ability to manage a cluster, but does not provide direct access to any configuration files. Any direct changes by an advanced user have to be done through CLI. Commonly used scenarios such as adding special arguments in configuration files are also done through CLI. In this chapter, we will look at the following topics:
• Proxmox Under the HoodThe Proxmox cluster directory structure Proxmox is a cluster-based hypervisor. It is meant to be used with several server nodes. By using multiple nodes in a cluster, we provide redundancy or High Availability to the platform while increasing uptime. A production virtual environment may have several dozens to several hundreds of nodes in a cluster. As an administrator, it may not be a realistic scenario to change configuration files in the cluster one node at a time. Depending on the number of nodes in a cluster, it may take several hours just to change one small argument in a configuration file of all the nodes. An administrator would have to go from node to node to apply these changes. To alleviate wastage of precious time, Proxmox implemented the clustered filesystem to keep all the configuration files, or any other common files shared by all the nodes in the cluster, in a synchronous state. Its official name is Proxmox Cluster file system or pmxcfs, which is a database-driven filesystem to store configuration files. Any changes made to any files or copied/deleted in this filesystem get replicated in real time to all the nodes using Corosync. Corosync Cluster Engine is a group communication system used to implement High Availability within an application. You can learn more about Corosync by visiting the link http://corosync.github.io/corosync/.
• Filename/location# /etc/pve/storage.cfg # /etc/pve/user.cfg
• Proxmox Under the HoodDissecting the configuration files We now know where all the important files that hold a Proxmox cluster together are placed. We will now go inside some of these files for better understanding of what they do and the command arguments they use. You can use any Linux editor to view/edit these configuration files. In this book, we will use #nano to view and edit. During the learning process, it will be a good idea to make a backup of the configuration files before editing them. In case something goes wrong, you will be able to replace it with the original working configuration file. Simply copy a configuration file using the following command:
• Chapter 2The <cluster>..</cluster> tag acts as the main body of the configuration file. All other syntaxes reside inside this tag. The beginning of this tag shows the name of the cluster as name="pmx-cluster" and the config file version number as config _version="2". This number represents how many times the file has been changed. Each change increases the version number incrementally: <cman keyfile="/var/lib/pve-cluster/corosync.authkey"></cman>
• Proxmox Under the HoodIt is possible to operate a cluster with just two nodes. But this form of setup is only good for very small usage, such as the home or test platform where downtime really has no effect on operation. The storage configuration file
• options vers=3 content images, vztmp1, backup, rootdir maxfiles 1Chapter 2 Almost all the settings in storage.cfg can be changed from the Proxmox GUI without using any CLI.
• Page 73Proxmox Under the Hood server 192.168.145.11 export /mnt/pmxnas01 options vers=3 content iso, vztmpl maxfiles 1 Now, we will explain the individual lines of the previous code.
• Chapter 2The following line shows the NFS-shared storage configured to hold an ISO image and OpenVZ/container template only: content iso, vztmpl maxfiles 1
• Proxmox Under the HoodThe following line shows the iSCSI extent stored on the iSCSI server, which could be backed up by a physical disk or a file. In our setup, we have used a file-based iSCSI extent: target iqn.2011-03.org.example.istgt:pmxtgt01
• Chapter 2User configuration files The user.cfg file holds all user and group information in the cluster and is located under /etc/pve/. It follows the following format to store all user information:
• Proxmox Under the HoodThe virtual machine configuration file The vmid.conf file stores configuration information for each virtual machine and is located under /etc/pve/nodes/<name>/qemu-server/<vmid.conf >. The folder structure divides all VM configuration files into categories based on nodes. For example, the configuration file for our VM with ID# 101 is stored in the following location:
• Chapter 2Almost all the options in this file can be set through the Proxmox GUI under the KVM virtual machine option menu tab. Some option values such as args, hotplug, and so on have to be added through CLI. The following is a chart of the possible options. The values can be used as virtual machine configurations: Possible values
• Proxmox Under the HoodDescription Options
• Page 80Options net (n): onboot:
• Proxmox Under the HoodOptions startup: Description
• Chapter 2Arguments in the KVM configuration file Arguments in a virtual machine configuration file is a way to extend the capability of the VM beyond just the default. For example, by default, sound is not enabled for a VM. In order to give a VM the ability to play audio/video, an argument has to be passed through the VM configuration file. The following are some examples of arguments that can be used in a Proxmox VM configuration file. Arguments can be added in the following format:
• Proxmox Under the Hood//Swap Allocated in bytes SWAPPAGES="0:1024M"
• SEARCHDOMAIN="domain.com"NAMESERVER="208.67.222.222" Chapter 2
• Proxmox Under the HoodMember nodes Located under /etc/pve/. members, the member node file shows all the member nodes that are a part of the Proxmox cluster. It is a great way to see the cluster status, when the Proxmox GUI becomes inaccessible for any reason. The following is the .members file in our basic cluster:
• The virtual machine list fileChapter 2 Located under /etc/pve/.vmlist, the virtual machine list file stores a list of all the virtual machines within the Proxmox cluster. The .vmlist file uses the following format to store the list:
• Proxmox Under the HoodSummary In this chapter, we looked at the location of important configuration files needed to run a healthy Proxmox cluster. We also looked through the configuration files from inside to have a better understanding at the parameters used and other possible values for different parameters. As mentioned earlier, most of these configuration files can be changed via a Proxmox GUI. But when the GUI becomes inaccessible for any reason, knowing where these files are located can save a tremendous amount of time by accessing them through CLI.
• Shared Storages with Proxmox3 In simple terms, shared storage is a medium to store data for simultaneous access by multiple devices or nodes in a network. As server and desktop virtualizations become the norm, shared storage today is much more critical for a virtual environment.
• Shared Storages with ProxmoxLocal storage versus shared storage Shared storage is not absolutely necessary in a Proxmox cluster environment, but without a doubt it makes storage management a walk in the park. In a small business with an extremely limited budget and IT staff, a local storage system will suffice. But when data grows beyond several terabytes and keeps growing exponentially every month due to virtual machine backups, shared storage starts to make a whole lot of sense. In most enterprise virtual environments with critical data, shared storage is the only logical choice due to the benefits it brings to the whole cluster operation. The following are considered benefits of using shared storage:
• Chapter 3LAN Switch node 03 node 02 node 04 node 01 vm21 vm22 vm31 vm32 vm 1 vm2 vm 11 vm 12 vm 10 vm30 vm40 vm 20 In the previous overly simplified diagram, there are four Proxmox nodes with 10 virtual machines on each. If node 01 needs to reboot, all the 10 virtual machines have to be stopped, the node needs to be rebooted, and then all the virtual machines must be powered up. If node 01 needs to be taken offline for hardware upgrade, then all virtual machines need to be stopped and we have to manually migrate each virtual machine to a different node. If node 01 fails completely, then all these 10 virtual machines will be inaccessible till node 01 is back on again. A brave administrator might just take the hard drives out of the failed node, plug them into another Linux machine, copy the virtual machine image files directly to another node, and then power them up. But it will not work if node 01 had anything but mirror RAID in it for all the hard drives.
• Shared Storages with ProxmoxIn the previous diagram, all 40 virtual machines are stored on a shared storage system. The Proxmox node only holds the configuration files for each virtual machine. In this scenario, if node 01 needs to be rebooted due to a security patch, all the virtual machines can be simply migrated to another node without powering down a single virtual machine. A virtual machine user will never notice that his or her machine has actually moved to a different node. If total Proxmox node failure occurs, the virtual machine configuration file can simply be moved from /etc/pve/ nodes/node01/qemu-server/<vmid>.conf to /etc/pve/nodes/node02/qemuserver/<vmid>.conf. Since all virtual machine configuration files are in a Proxmox-clustered filesystem, they can be accessed from any node. Refer to Chapter 2, Proxmox Under the Hood, for Proxmox cluster filesystem or pmxcfs. With virtual machine image files on shared storage, Proxmox migration does not have to move all the image files using rsync from one node to another, which allows much faster virtual machine migration. The rsync is an open source program and network protocol for Unix-based systems. It provides nonencrypted or encrypted incremental file transfers from one location to another.
• Chapter 3In an enterprise environment, storage space should be added seamlessly without shutting down critical nodes or virtual machines. By using a shared storage system, virtual machines can now go beyond one node to scattered multiple nodes spanned across regions. Shared storage such as distributed filesystem (for example, Ceph) can span across several racks and comprise well over several petabytes of usable storage space. Simply add a new node with a full bay of drives and tell the storage cluster to recognize the new node to increase storage space for the entire cluster. Since shared storage is separated from the virtual machine host nodes, storage can be increased or decreased without disturbing any running virtual machines. Later in the book, we will see how the Ceph-distributed filesystem can be expanded as much as your budget or needs allow you to. Centralized backup
• Shared Storages with ProxmoxCentral storage management By separating shared storage clusters from primary Proxmox clusters, we can manage two clusters without them interfering with each other. Since shared storage systems can be set up with separate nodes and physical switches, managing them based on different authorizations and permissions becomes an easier task. NAS, SAN, and other types of viable shared storage solutions come with their own management programs, from where an administrator or operator can check storage cluster health, disk's online/offline status, storage availability, and so on. The Ceph storage itself does not have a graphical interface for management. The following screenshot is what a typical Ceph command-line status looks like in order to check all the attached nodes and health of the storage cluster:
• Chapter 3FreeNAS Services Plugins
• Shared Storages with ProxmoxLocal storage Limited to available drive bays of a node
• Chapter 3The following table is a brief summary of the different image formats and their possible usage: Image
• Shared Storages with ProxmoxThin provisioning is when the virtual disk image file does not preallocate all the blocks, thus keeping the size of the image file to only what we want. As more data is stored in the virtual machine, the thin provisioned image file grows till it reaches the maximum size allocated. Thick provisioning, on the other hand, is when the virtual disk image file preallocates all the blocks, thus creating an image file which is exactly of the size set prior to creating the virtual disk image file. The .qcow2 format also has very high I/O overhead due to its additional software layer. Thus, it is a bad choice of image format for a VM such as a database server. Any data being read or written into the image format goes through the qcow2 software layer, which increases the I/O making it slower. A backup created from a qcow2 image can only be restored to an NFS or local directory.
• Bus/device typeMaximum allowed IDE
• Shared Storages with ProxmoxWhen a disk image file is selected, the Resize disk and Move disk buttons become available. Virtual machine image files can also be manipulated using CLI commands. The following are a few examples of the most common commands to delete, convert, and resize an image file: Usage: qemu-img command [command options] #qemu-img create -f <type> -o <filename> <size> #qemu-img create -f raw test.raw 1024M
• Page 100For Linux VM with QCOW2 partition, use the following command lines: # apt-get install nbd-client
• Shared Storages with ProxmoxIn the Move disk option menu, just select the Target Storage and Format type, and then click on Move disk to move the image file. Moving can be done live without shutting down the VM. Checking Delete source will delete the source image file after the move is complete. The Format type in the Move disk option will be greyed out if the destination storage only supports one image format type. In the previous example, ssd-ceph-01 is an RBD storage in Ceph Pool. Since RBD only supports the RAW format, the format type has been greyed out automatically. Learn more about Ceph in Chapter 7, High Availability Storage for High Availability Cluster.
• DirectoryChapter 3 Directory storage is mostly a mounted folder on the Proxmox local node. It is mainly used as local storage. By default, this location is mounted under /var/lib/vz.
• Shared Storages with ProxmoxRADOS Block Device RADOS Block Device or RBD storage is provided by Ceph distributed storage system. It is the most complex storage system, which requires multiple nodes to be set up. By design, Ceph is a distributed storage system and can be spanned over several dozen nodes. The RBD storage can only store raw image formats. To expand a Ceph cluster, simply add a hard drive or a node and let Ceph know about the new addition. Ceph will automatically rebalance data to accommodate the new hard drive or node. Ceph can be scaled to several petabytes or more. Ceph also allows multiple pool creations for different disk drives. For example, we can store database servers' VM images on an SSD-driven pool and backup server images on a slower- spinning drive pool. Ceph is the recommended storage system for medium to large cluster environments due to its resiliency against data loss and the simplicity of storage expandability.
• optionsNoncommercial/commercial storage Chapter 3
• Shared Storages with ProxmoxTrade-off of noncommercial open source solutions is the money that is saved, which usually gets exchanged with the time spent on research and mistakes. In this book, we will look at two noncommercial storage options, FreeNAS and Ceph. Both are stable enough to set up a fully functional Proxmox cluster. FreeNAS is more targeted for small businesses or home lab usage, while Ceph is capable enough to take on just about any simple or complex cluster environment. There are some scenarios where FreeNAS has been used as backup servers to back up virtual machines. Multiple FreeNAS servers can be set up to perform separate backup tasks.
• Page 106Chapter 3 FreeNAS provides many services such as AFP, NFS, CIFS, iSCSI, FTP, SSH, SNMP, RSYNC, and so on. FreeNAS can take advantage of ZFS, which offers a stable platform to build a virtual environment on. ZFS was originally designed by Sun Microsystems combining filesystem and logical volume manager to create a new type of filesystem, which supports high storage capacities, provides protection against data corruption, performs continuous integrity checking, and so on. To know more about ZFS, visit http://en.wikipedia.org/wiki/ZFS.
• Shared Storages with ProxmoxFreeNAS also supports advanced networking features such as network bonding to increase storage node bandwidth. The only drawback of noncommercial FreeNAS is that it does not have High Availability or an out-of-the-box clustering option. With some work, it is possible to set up High Availability FreeNAS cluster using heartbeat, but it is not stable to be used in a production environment. But a setup of two replicated non-High Availability FreeNAS nodes can be easily achieved using RSYNC built-in with FreeNAS. For those who really want to try a shared storage such as FreeNAS but with availability, give NAS4Free (http://www.nas4free.org) a try. By following the instructions at http://blackcatsoftware.us/inprogress-configure-nas4free- high-availabilty-storage-carphast zfs/, High Availability shared storage can be created with NAS4Free.
• Page 108Chapter 3 The most common usage of FreeNAS with the Proxmox cluster is for backup purpose. Since backup nodes do not need to be high performing nodes and downtime is tolerable, FreeNAS is an excellent choice for this task. With two FreeNAS backup nodes, VM backups can be replicated to each FreeNAS node and provide backup redundancy. NFS share is most commonly used for a Proxmox VM backup. FreeNAS has an excellent built-in reporting system, which shows minute-by-minute information of different critical parts of the FreeNAS node. Data is presented in a very easy way to understand the graphical format. Information about CPU, memory, storage quota, network bandwidth, and so on can prove very valuable when there is an issue. Sometimes, this data can also prevent disasters by spotting symptoms such as out of memory or high CPU consumption continuously. The following screenshot is an example of the Reporting system showing CPU Usage and System Load:
• Shared Storages with ProxmoxSummary In this chapter, we took a look at what storage options are supported by Proxmox and their advantages and disadvantages. We also saw the types of virtual image files that can be used with Proxmox and when to use them. Storage is an important component for Proxmox clustering. Since a storage system is where virtual machines are created and operated from, a properly implemented storage system is very critical to make any cluster a successful one. With proper planning of different storage needs and choosing the right format and option, a lot of hassle and frustration can be minimized later on. In the previous chapter, we very briefly saw the arguments usage in the Proxmox configuration files. In the next chapter, Chapter 4, A Virtual Machine for a Virtual World, we will analyze those arguments in greater detail to extend the cluster ability beyond just its default. Using arguments, we can enable features such as sound, USB, backup performance, and so on for a virtual machine.
• A Virtual Machine for a Virtual World4 We are about to embark upon a new kind of virtual world, which is beyond just basic. Through previous chapters, we have familiarized ourselves with Proxmox up close and personal. We saw what a Proxmox GUI looks like and got our hands dirty with the Proxmox configuration files and directory structure. We also learned what shared storage system is and how it is integrated with the Proxmox cluster. In this chapter, we are going to take it one step further by covering at the following topics:
• A Virtual Machine for a Virtual WorldCreating a VM from a template Let us start by creating our second virtual machine from the Ubuntu template we created in the Transforming VM into a Template section in Chapter 1, Dive into the Virtual World with Proxmox. Right-click on the template and select Clone, as shown in the following screenshot:
• Chapter 4Advanced configuration options for a VM We will now look at some of the advanced configuration options we can use to extend the capability of a KVM virtual machine.
• A Virtual Machine for a Virtual WorldWith default options added during the virtual machine creation process, the following code is what the VM configuration file looks like: ballon: 512 bootdisk: virtio0 cores: 1 ide2: none, media=cdrom kvm: 0 memory: 1024 name: pmxUB01 net0: e1000=56:63:C0:AC:5F: 9D, bridge-vmbr0 ostype: 126 sockets: 1 virtio0: vm-nfs-01:102/vm-102-disk-1.qcow2, format=qcow2, size=32G
• Chapter 4The modules can also be loaded automatically during the virtual machine boot by inserting them in /etc/modules. Simply add acpiphp and pci_hotplug on two separate lines in /etc/modules. Adding virtual disk/vNIC
• A Virtual Machine for a Virtual WorldOne very important thing to keep in mind is that a nested virtual environment will have a significantly lower performance than a real virtual environment. If the nested virtual environment also has virtualized storage, performance will degrade significantly. The loss of performance can be offset by creating a nested environment with an SSD storage backend. When a nested virtual environment is created, it usually also contains virtualized storage to provide virtual storage for nested virtual machines. This allows for a fully isolated nested environment with its own subnet and virtual firewall. There are many debates about the viability of a nested virtual environment. Both pros and cons can be argued equally. But it will come down to the administrator's grasp on his or her existing virtual environment and good understanding of the nature of requirement. The cluster that we have been building through out this book so far is a nested virtual cluster itself, specifically created to write this book. This allowed us to build a fully functional Proxmox cluster from the ground up without using additional hardware. The following screenshot is a side-by-side representation of a nested virtual environment scenario:
• Chapter 4Our nested cluster is completely isolated from the rest of the physical cluster with a separate subnet. Internet connectivity is provided to the nested environment by using a virtualized firewall 1001-scce-fw-01. Like the hotplugging option, nesting is also not enabled in the Proxmox cluster by default. Enabling nesting will allow nested virtual machines to have KVM hardware virtualization, which increases the performance of nested virtual machines. To enable KVM hardware virtualization, we have to edit the modules in /etc/ of the physical Proxmox node and <vmid>.conf of the virtual machine. We can see that the option is disabled for our cloned nested virtual machine in the following screenshot:
• A Virtual Machine for a Virtual World3. After the Proxmox nodes reboot, add the following argument in the <vmid>. conf file of the virtual machines used to create a nested virtual environment: args: -enable-nesting 4. Enable KVM hardware virtualization from the virtual machine option menu through GUI. Restart the nested virtual machine.
• Page 118Proxmox backup and snapshot options Proxmox has the following two backup options: ⚫ Full backup: This backs up the entire virtual machine.
• A Virtual Machine for a Virtual WorldProxmox 3.2 and above cannot do full backups on LVM and Ceph RBD storage. Full backups can only occur on local, Ceph FS, and NFS-based storages, which are defined as backup during storage creation. Please note that Ceph FS and RBD are not the same type of storage even though they both coexist on the same Ceph cluster. In Chapter 7, High Availability Storage for High Availability Cluster, we will create both RBD and Ceph FS in a Ceph cluster. The following screenshot shows the storage feature through the Proxmox GUI with backup-enabled attached storages: Datacenter
• Chapter 4Create: Backup Job Send email to:
• A Virtual Machine for a Virtual WorldStart Time Unlike Day of Week, only one time slot can be selected. Multiple selections of time to backup different times of the day are not possible. If the backup must run multiple times a day, create a separate task for each time slot.
• CompressionChapter 4 By default, the LZO compression method is selected. LZO (http://en.wikipedia. org/wiki/Lempel-Ziv-Oberhumer) is based on a lossless data compression algorithm, designed with the decompression ratio in mind. LZO is capable to do fast compression and even faster decompressions. GZIP will create smaller backup files at the cost of high CPU usage to achieve a higher compression ratio. Since higher compression ratio is the main focal point, it is a slow backup process. Do not select the None compression option, since it will create large backups without compression. With the None method, a 200 GB RAW disk image with 50 GB used will have a 200 GB backup image. With compression turned on, the backup image size will be around 70-80 GB.
• A Virtual Machine for a Virtual WorldThe actual snapshot creation process is very straightforward. Just enter a name, select the RAM content, and type in some description. The Name textbox does not allow any spaces and the name must start with an alphabet as shown in the following screenshot: VM 102: Take Snapshot
• Chapter 4Snapshots are also a great way to test new software or configuration on a virtual machine. Take a snapshot before installing a software or applying new configuration. After the software test or if the configuration does not work as intended, simply roll back to the previous snapshot. This is much faster and cleaner instead of uninstalling the tested software itself. Deleting old backups
• A Virtual Machine for a Virtual WorldFor higher backup redundancy and assurance, we can attach multiple shared storages to store backups. In the example shown in the following screenshot, there are four separate shared storages attached for daily, weekly, monthly, and yearly backup. There are also two backup tasks scheduled: one for daily backup, which keeps six days of backups, and the second task for weekly backup, which keeps four backups The following screenshot shows the Backup tab: Datacenter pmxvm01 pmxvm02
• Edit: NFSID: backup-weekly
• A Virtual Machine for a Virtual WorldCommand-line vzdump The entire backup process can be handled from the command line in case the GUI becomes inaccessible. The command to start a backup is as follows:
• Page 128For a complete list of options for vzdump, qmrestore, and vzrestore, visit the following pages: http://pve.proxmox.com/wiki/Vzdump_manual http://pve.proxmox.com/wiki/Qmrestore_manual
• A Virtual Machine for a Virtual World#lockwait The Proxmox backup uses a global lock file to prevent multiple instances running simultaneously. More instances put extra load on the server. The default lock wait in Proxmox is 180 minutes. Depending on different virtual environments and number of virtual machines, the lock wait time may need to be increased. If the limit needs to be 10 hours or 600 minutes, adjust the option as follows:
• if ($phase eq 'job-start' || $phase eq 'job-end'$phase eq 'job-abort') { my $dumpdir $ENV{DUMPDIR};
• A Virtual Machine for a Virtual World# example: copy resulting log file to another host using scp if ($phase eq 'log-end') { #system ("scp $logfile backup-host: /backup-dir") == 0 || # die "copy log file to backup-host failed";
• SummaryChapter 4 In this chapter, we have covered some interesting topics, such as nested virtual environment, the hotplug feature, and Proxmox backup/restore. A good backup plan can save the day many times over. Although Proxmox does not provide everything you need for backup such as a granular file backup, to back up a virtual machine is very helpful. Backup features in the Proxmox platform have proven to be reliable in production environments and during actual disaster scenarios.
• Page 133
• Page 1345 Network of Virtual Networks
• Network of Virtual NetworksIntroduction to a virtual network A virtual network is a software-defined network where all links and components may or may not have direct interaction with physical hardware. In most cases, direct interaction with physical hardware is made by the hypervisor or the host controller. All links between virtual machines, virtual switches, virtual bridges, and virtual network interfaces are made completely virtually. The following are the two types of network virtualization:
• Chapter 5Physical network versus virtual network Let us look at the following diagrams to see the difference between a physical network and a virtual network. The following diagram represents a physical network without any virtualization platform:
• Network of Virtual NetworksDo not worry if the virtualization diagram is overwhelming at first glance. We will learn about all the components used in both diagrams later on. Both diagrams are in basic form. Some non-relevant components may have been omitted to keep the diagrams simpler. Before we dive into virtual network building blocks, we need to understand how networks are set up in the preceding diagrams. Both the diagrams represent the same office setup where the main administrative department is on the second floor, and the accounting department is on the fourth floor of the building. It is apparent from the diagrams that a physical network is undoubtedly simpler than a virtual network, but by leveraging virtualization, we can cut cost, increase efficiency, reduce hardware maintenance complexity, and increase portability.
• Chapter 5In this setup, all servers and user machines are virtualized. If there is a need for a new server, it is just a matter of creating a virtual server with vNIC with a few clicks. Since Proxmox as a hypervisor has redundancy by design, a dead physical server node is no longer an issue. In such a scenario, all virtual machines can simply be migrated to another available Proxmox node, and everything is up and running in minutes. Both the departments are separated by two virtual bridges. Through the use of the Proxmox GUI, all management can be done from one location, including backup and restore. Virtual servers can be sent to another remote company location through WAN. Although a virtual network setup is much more robust and feature-rich, it has a much lower budget requirement. New departments can be added by creating new virtual bridges for separate subnets and using VLANS on existing physical network switches. Networking components in Proxmox
• Network of Virtual NetworksIntel e1000 is a Linux kernel driver to virtualize Intel architecture-based virtual network interfaces. This is the default vNIC for new virtual machines in Proxmox since it is supported by all major operating systems, such as Linux, Windows, and Mac OS X, without needing additional drivers. Proxmox has four models of virtual network interfaces: Intel e1000, VirtIO, Realtek RTL8139, and VMware vmxnet3. It should be noted that out of these four models, VirtIO provides the maximum network performance for a VM. All Linux-based operating systems come equipped with VirtIO drivers. For Windows, the VirtIO interface driver can be downloaded from http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_ Drivers. For Mac OS, the VirtIO interface driver can be downloaded from https://github.com/pmj/virtio-net-osx. Virtual bridge
• Chapter 5Once VLAN data leaves the virtual environment, a physical network switch with the VLAN feature tags each data with an ID and then directs the data to its proper destination. Each subnet should have the same VLAN ID on the virtual environment and on the physical network switch. VLAN helps reduce the broadcast traffic of multiple domains on the same network. By segmenting a large network into smaller VLAN, broadcasts can be sent only to relevant VLAN without interrupting other data traffic on the network. Imagine a 10-lane highway connecting point A to point B. Each lane is designated to one manufacturing plant that needs to transport goods from point A to point B. Plant 1 is designated to lane number 1 (VLAN ID 1); plant 2 is designated to lane number 2 (VLAN ID 2); and so on. Even though all plants share the same highway, they each have their own lane not shared by others. VLAN follows the same logic. VLAN also provides an added security layer on a multidomain network since a user can no longer just plug into the network and capture just about any data of any domains on the network. Network segmentation is usually done with a layer 3 device such as a router. But using VLAN, significant cost saving can be achieved with an already-existing layer 2 device on the network such as a switch. There are seven layers defined by the Open Systems Interconnection (OSI) model by which network communication takes place. For in-depth details on OSI, follow the link http://en.wikipedia.org/wiki/OSI_model.
• Network of Virtual NetworksNetwork bonding Network bonding or Teaming or Link aggregation is a concept where multiple interfaces are combined to increase throughput, set up network redundancy, and balance network load. This concept is heavily used in high-demand environments where downtime and slow network I/O are not acceptable. The Proxmox GUI provides the excellent feature of creating and managing bonding within the cluster node. Bonding modes supported by Proxmox are balance-rr, active-backup, balance -xor, broadcast, Link Aggregation Control Protocol (LACP), or 802.3ad, balance-tlb, and balance-alb. The following table lists the various bonding modes as well their policies and descriptions:
• Bonding ModePolicy Description balance-tlb
• Network of Virtual NetworksNetwork configuration file We are now going to a look at how a Proxmox network configuration file works and how we can modify it to adapt to different networking environments. This configuration file is stored at /etc/network/interfaces. Network configuration can be modified both from the GUI and the CLI. Any changes made from the GUI are saved temporarily in /etc/network/interfaces.new. The changes get transferred to /etc/network/interface only after reboot. You can also directly edit the interfaces file at /etc/network/ and activate or deactivate components with #ifup <name> or #ifdown <name> without needing to reboot. All virtual bridges must be configured on all the Proxmox nodes in the cluster. If there is a lot of bridge configurations in the interface file, simply copy them to all nodes and modify only the physical interfaces on each node. The following screenshot is the GUI representation of the configuration file for node pmxvm01:
• Chapter 5Based on this configuration, the pmxvm01 node has two virtual bridges set up. Bridge vmbro is tied to physical network interface eth0, and bridge vmbrl is tied to eth1 of the node. If we want to create a fully isolated subnet without any outgoing traffic, we could edit bridge vmbrl to look like the following code: auto vmbrl iface vmbrl inet manual bridge ports none bridge_stp off bridge_fd 0 bridge_stp
• Network of Virtual NetworksFor the most part, the default value of 0 is enough. In a very complex virtual environment with several dozen bridges, increasing this number to 3 or 4 might help. Without this delay, the bridge will start transmitting data packet regardless of whether the other destination bridge is available or not. Increasing the delay time allows the source bridge to check all bridges and not transmit any data if the destination bridge is down, thus preventing unnecessary network bandwidth consumption. There are many more bridge_options to be used in the network configuration file, such as bridge_hello, bridge_maxage, bridge_bridgeprio, and so on. Bridge options are Linux- specific and beyond the topic of this book. For in-depth information on bridges, visit http://www.linuxfoundation.org/ collaborate/workgroups/networking/bridge.
• Chapter 5The new virtual bridge vmbr200 should now be activated and running. If there are more cluster nodes, then the steps from 1 to 5 must be completed on all the nodes. The /etc/network/interfaces configuration file should look as follows: auto lo iface lo inet loopback auto vmbro iface vmbr0 inet static address 192.168.145.1 netmask 255.255.255.0 gateway 192.168.145.254 bridge ports eth0 bridge stp off bridge_fd 0.
• Network of Virtual NetworksAdding a bonding interface We will now see network bonding in our cluster. There are several types of bonding options available. But only balance-rr, active-backup, and LACP (802.3ad) are most widely used. The balance-rr option provides the round robin method to increase overall interface bandwidth with failover. The balance-rr option does not require any special network switch. Just about any switch can be used to make this work. The major drawback of balance-rr is waste of data packets. LACP is known as industry quality bonding.
• Chapter 5In our cluster, we are now going to set up LACP bonding for the bridge vmbr1. Later in the book, we will set up an enterprise-standard distributed shared storage system using Ceph. We will use the bridge vmbrl to connect the shared storage with the Proxmox cluster. Let's start by adding a third network interface card on each node if you have not done so. So, both nodes now should have three physical network interface cards on each. Perform the following steps to configure LACP or 802.3ad network bonding: 1. Securely log in to the node pmxvm01.
• Network of Virtual Networks8. Set up LACP or link aggregation on a physical switch. The following screenshot is an example of LACP setting on Netgear GS724T 24 port switch: GS724T
• Page 150netmask 255.255.255.0 bridge ports bond0 bridge_stp off bridge_fd 0 auto vmbr200 iface vmbrl inet static bridge_ports none bridge_stp off bridge_fd 0 Chapter 5
• Network of Virtual Networks bond_miimon X bond_updelay XThe following several options can be used while configuring bonding: X = integer number. It sets the MII link monitoring frequency in milliseconds. This option checks the frequency at which each slave port is inspected for link failures.
• Adding VLANChapter 5 VLAN can be set up on both virtual machines and bridges. If the VLAN traffic leaves the virtual environment, it is important to set VLAN on physical network switch by configuring port trunking and VLAN tagging. Tagging VMs with a VLAN ID is very straightforward through the Proxmox GUI. Just enter the VLAN ID when adding a network interface to the VM or edit already added vNICs. The following screenshot shows the dialog box to edit a network device through the Proxmox GUI:
• Network of Virtual Networks3. Add the following lines at the beginning of the file: auto bond0.10 iface bond0.10 inet manual vlan-raw-device bond0
• Page 154Our final network interface configuration should look as follows: auto lo iface lo inet loopback
• Page 155Network of Virtual Networks bridge ports none bridge_stp off bridge_fd 0 auto vmbr10 iface vmbrl inet static bridge ports bond0.10 bridge_stp off bridge_fd 0 A good practice to identify which VLAN belongs to which bridge is to use the same numeric number for both the interfaces. For example, a bridge vmbr10 will have the same VLAN ID 10. Without some order in the beginning, bridges and VLANs will quickly get out of control as network grows over time.
• Chapter 5Physical Switch Internet
• Network of Virtual NetworksFor firewall virtualization, pfsense is a great choice to set up. It is easy to set up, yet extremely powerful and customizable. Get pfsense and more information from www.pfsense.org. The following diagram is an example of a multitenant virtual environment:
• Chapter 5Of course, the following diagram is a very simplified form of the actual network topology of an academic institution. But the basics of using VLANs and bridges are the same for any network size. Remote Campus
• Network of Virtual NetworksIn this type of setup, the service provider hosts or "rents out" computing time and storage space to their clients. Because of standard monthly subscription or SLA-based payment methods required for this type of service, the term multitenancy quickly gained popularity. Basically, a multitenant virtual environment is where several isolated networks coexist on the same platform without interfering with one another. Almost all public datacenters are multitenancy platforms. Multitenancy is not new in the world of information. The first multitenant environment appeared back in the 1960s, when companies rented processing time and storage space on mainframe computers to reduce giant expenses of mainframe operation. The virtual environment only augmented the same idea exponentially by leveraging all the virtualization features Proxmox provides. By combining virtualization with cloud computing, multitenancy is able to get a very strong footing to serve better and to more customers without increasing financial overheads. Prior to virtualization, the physical space and power requirements to host customers in an IAAS environment meant it was rare and very expensive, thus not many people enjoyed its benefit.
• Chapter 5Small Software Company Company Company Small
• Network of Virtual NetworksOn the client side, they only have simple desktop computers and mobile devices to access their virtual cloud resources, such as desktop, storage, processing power, and so on. Clients access these resources through a virtual means, such as Virtual Network Computing (VNC), SPICE, or Remote Desktop Protocol (RDP). Virtual networks are isolated with separate subnets. VLANs are set up (not shown in the diagram) to reduce mass broadcast traffic. All virtual machine data is stored on a separate storage cluster with full redundancy. A backup cluster does regular backup of all virtual machines, and granular file backup with histories are done with a third-party backup software. A virtual firewall cluster is set up in between the virtual environment and the host Ethernet interface to provide Internet connectivity to all client virtual machines. Each virtualized firewall has several vNICs to connect to each subnet. A typical virtual firewall with multiple vNIC would look as shown in the following screenshot:
• Chapter 5Since the firewall is virtualized, we can add any number of virtual network interfaces without worrying about running out of physical slots. A virtualized clustered firewall provides maximum uptime. Each company network in this example has its own virtual bridge, which only talks to that company's virtual machines and firewall interface, eliminating any chance of packet sniffing by other company networks. Packet sniffing is a process when data packets passing through a network interface are captured and analyzed. A packet sniffer software can be placed in a subnet to capture data. This is common practice of someone with malicious intention to capture sensitive unencrypted data passing through such as usernames and passwords in clear texts.
• Network of Virtual NetworksProxmox provides all the tools we need to build any level of virtual network. It comes down to the network administrator's imagination, the company's budget, and the need to foresee how all pieces should come together to form a well-designed and efficient virtual network. The best part is that any mistake is easily correctable in a virtual environment. We can always go back and change things until we are satisfied. For this very reason a virtual network is always evolving. Over time, a virtual network becomes an extension of the network administrator's mind network. The configurations and design of a virtual network infrastructure can give us a window into how that administrator thinks and the logic they used to construct the environment. In the next chapter, we are going to learn the High Availability feature of Proxmox, which provides automatic redundancy to the Proxmox virtual environment. To reduce downtime, it is important to have a functional High Availability to automigrate virtual machines to healthy nodes during any incident. We will look at some configuration files to enable High Availability in our cluster.
• Proxmox HA- ZeroDowntime 6
• Proxmox HA - Zero DowntimeIn a small environment this can be manageable, but in a large environment of hundreds of virtual machines and nodes, constant monitoring is just not realistic. Although there can be monitoring software in place to automatically alert administrators about any node failure, without HA the administrator will have to manually move or migrate any virtual machine from a faulty node. This can cause longer downtime due to human response time. HA takes the longer response time out of the equation by simply moving or migrating virtual machines to a node as soon as server hardware failure occurs. High Availability in Proxmox
• FencingChapter 6 Fencing is a concept of isolating a node or its resources during node failure so that other nodes cannot access the same resources causing data corruption. In Proxmox, fencing prevents multiple nodes from running on the same virtual machine or cluster-specific services. Without fencing, Proxmox HA cannot be set up. Fencing ensures data integrity during a failure by preventing all nodes from running on the same virtual machine or cluster services at the same time. There are several ways to set up fencing such as the following:
• Proxmox HA- Zero DowntimeResource-level fencing prevents the same cluster-related services or resources from starting on all nodes in the cluster simultaneously. Node-level fencing makes sure that the node in question does not run any services at all by simply power cycling the node with the help of a managed PDU or IPMI. This method is also known as STONITH or Shoot The Offending Node In The Head. There are several types of fencing devices, such as a network-managed switch, APC master switch, Dell drac card, and so on. In this chapter, we are going to use APC-managed PDU AP7921 to set up device-based fencing. AP7921 is completely tested with Proxmox and very much dependable. Also, setting up fencing with PDU is very simple. If you would like to try setting up fencing using other methods, visit the Proxmox Wiki at https://pve.proxmox.com/wiki/Fencing. We can summarize the requirements of device-based fencing and HA setup as follows:
• Chapter 6To make sure that this feature is truly enabled, try to unplug the power cord and then plug it back in to see if the node powers up. Without this feature enabled, the node cannot be power cycled by the fencing device. Creating an APC-managed PDU user
• Proxmox HA - Zero DowntimeWe can create the new user from Outlet Manager under the Device Manager tab. Fill in the information as shown in the following screenshot and then simply hit Apply: Home Device Manager Logs Administration
• Task viewer: HA 101 - StartOutput Status Stop
• Proxmox HA- Zero DowntimeThe fence_tool program is in the Linux operating system. It is used to join or leave nodes in the default fence domain. It also starts the fence daemon fenced. Even though fenced can be started or stopped without fence_tool, this small tool takes some added measures that are very helpful in a production environment. To get more information on fence_tool and other options that can be used, visit http://linux.die.net/man/8/fence_tool. 4. We are now going to edit the /etc/pve/cluster.conf file to add a fencing device. We can edit the file from the CLI and activate it from the GUI. We must remember to increase the config_version number when we make any changes to the cluster.conf file. This ensures that all the nodes will apply the new settings. Our current cluster.conf file looks as follows: <?xml version="1.0"?>
• Chapter 6Before we start editing the cluster.conf file, we are going to copy the file and edit the copied file. That way we can activate it from the GUI. The command to do so is as follows: #cp /etc/pve/cluster.conf /etc/pve/cluster.conf.new
• Proxmox HA-Zero Downtime</clusternode> <clusternode name="pmxvm02" nodeid="2" votes="1"> <fence>
• DatacenterSearch Summary Backup
• Proxmox HA - Zero DowntimeWe can enable the HA feature for a virtual machine by navigating to Datacenter | HA, as shown in the following screenshot: Server View
• Chapter 6In the previous screenshot, rm simply means resource manager. We can see that the VM 101 is now part of Proxmox HA. Any virtual machine we add in HA will show under the rm segment of cluster.conf. We can also see the virtual machine's HA status in the Summary screen, as shown in the following screenshot:
• Proxmox HA- Zero DowntimeRecovery of the node will continue only after the command has been entered. Manual fencing requires user intervention and should not be used in a production environment.
• SummaryChapter 6 In this chapter, we have covered an important feature named Proxmox High Availability. It is a feature that gives control to Proxmox nodes to automigrate virtual machines during a node failure or maintenance. We also saw how this feature actually works and what are the steps involved in setting it up. We learned about new equipment called managed PDU, which allows an administrator to power up or power down an equipment remotely. We also configured one of our virtual machines to take advantage of High Availability.
• Page 179
• Page 1807 High Availability Storage for High Availability Cluster
• High Availability Storage for High Availability ClusterIntroducing the Ceph storage Ceph is a distributed storage system designed keeping performance, reliability, and scalability in mind. Performance, reliability, and scalability are three keys for any enterprise-class network environment, and Ceph provides just that. As the capacity of Ceph cluster grows with the addition of new hard drives or solid-state drives, performance grows with it. This is simply because when user data is sent to a Ceph cluster, it is divided into a number of pieces depending on the disk drives and is written simultaneously by all the disk drives. Of course, this is an overly simplified explanation of the entire process. We will look into Ceph mechanisms later in this chapter. Ceph's built-in self-healing system provides outstanding reliability, and the simplicity of adding disk drives or a physical node in the cluster to increase overall capacity provides unprecedented scalability. A Ceph cluster provides the following three storage types:
• FilesystemChapter 7 Ceph Filesystem sits on top of Object Storage and provides a POSIX-compliant filesystem to store files and folders such as a disk drive. The Filesystem storage type uses a MetaData Server (MDS), which maps files and directories stored in the Filesystem to the objects stored in Object Storage. MDS can also rebalance data stored in Filesystem among cluster hosts, ensuring high performance and preventing higher load on a specific host. Loss of MDS will cause loss of data. Ceph FS can be integrated with Proxmox by mounting a local directory on the Ceph FS pool.
• High Availability Storage for High Availability ClusterFromax Virtual Environment Version: 3.2-1/1933730b PROXMOX
• Page 184Maps Chapter 7
• High Availability Storage for High Availability ClusterCRUSH map Controlled Replication Under Scalable Hashing (CRUSH) is an algorithm used in Ceph to store and retrieve data by computing data storage locations within the cluster. It does so by providing a per-device weight value to distribute data objects among storage devices. The value is autoassigned based on the actual size of the disk drive being used. For example, a 2 TB disk drive may have the approximate weight of 1.81. The drive will keep writing data until it reaches this weight. By design, CRUSH distributes data evenly across weighted devices to maintain a balanced utilization of storage and device bandwidth resources. A user can customize a CRUSH map to fit any cluster environment of any size.
• Page 186OSD Chapter 7
• Page 187MDS High Availability Storage for High Availability Cluster
• Chapter 7For better efficiency, we recommend a total of 50-100 PGs per OSD for all pools. Each PG will consume a certain number of node resources, such as CPUs and memory. A balanced distribution of PGs ensures that all nodes and OSDs in the nodes are not out of memory or that the CPU does not face overload issues. A simple formula to follow while allocating PG for a pool is as follows: Total PGS=(OSD×100) / Number of Replicas
• High Availability Storage for High Availability ClusterFor instance, in the example of 150 OSDs, three Ceph nodes, and two replicas, our PG was 7500 per pool. This gave us 50 PGs per OSD. If we had three pools in that setup and each pool had 7500 PGs, then the total number of PGs would have been 150 per OSD. To balance the PGs across the cluster, divide 7500 by 3 for three pools and set a PG of 2500 for each pool. This gives us 2500/150 OSDs = 16 PGs per pool per OSD or 16 x 3 pools = 48 total PGs per OSD. Since we have two replicas in this setup, the final total PGs per OSD will be 48 x 2 replicas = 96 PGs. This is within the recommended 50-100 range of PGs per OSD. Here is another example scenario. Try to see if you can come up with the final PG value on your own. The answer is given after the scenario. The calculation for this scenario is given at the end of this chapter.
• Proxmox Network192.168.1.0/24 Firewall
• Node typeOSD = 2 nodes MON 3 nodes
• Chapter 7We will try our best to look into all aspects of the Ceph installation in this chapter. However, if we have missed out on something, Ceph's official documentation may be a good start for additional information. For the official Ceph documentation, visit http://www.ceph.com/ docs/master/start/intro.
• Page 193High Availability Storage for High Availability Cluster dns-nameservers 208.67.222.222 # sudo nano /etc/hostname ceph-admin-01
• Page 1944. Define zones for the local domain using the following command: # sudo nano /etc/bind/named.conf.local Enter the following entries in the named.conf.local file:
• High Availability Storage for High Availability Cluster4W ; expire 1D ; minimum
• Page 1965 6
• High Availability Storage for High Availability ClusterUpdating Ubuntu #sudo apt-get update
• Chapter 7As of this writing, the codename of Ceph's latest release is Firefly. Before installing ceph-deploy, be sure to check Ceph's latest codename. Change the name in the echo command accordingly: # echo deb http://ceph.com/debian-<ceph_codename>/ $ (1sb_release -sc) main sudo tee /etc/apt/sources.list.d/ceph.list
• Page 199High Availability Storage for High Availability Cluster mon_host = 192.168.10.11, 192.168.10.12,192.168.10.13 auth_cluster_required cephx cephx auth_service_required = cephx auth_client_required filestore_xattr_use_omap = true
• Page 200Complete details of the Ceph configuration or other files are beyond the scope of this book. A detailed documentation is available at http://ceph.com/docs/master/. Chapter 7
• High Availability Storage for High Availability ClusterThe Ceph cluster creates a default user named admin for cluster management during the cluster creation process. The user admin uses ceph.client.admin.keyring for all administrative tasks in a Ceph cluster. Let's check our progress so far using the keys we just gathered. Simply run the following command to check the cluster status:
• Chapter 7Before we create OSDs, we need to know the drives available in the nodes and the drive names. The following command format will give us a list of drives for a node in the cluster: # ceph-deploy disk list <node_name>
• High Availability Storage for High Availability ClusterAs soon as the OSDs are created, the Ceph cluster will achieve the active+clean status for the PGs. We can verify the status of the cluster with the following command: #sudo ceph-s cluster 472cefbe-11a6-40f7-a352-5d245ae89fb3 health HEALTH OK monmap e2: 3 mons at {ceph-mon-01-192.168.10.11:6789/0, ceph-mon- 02-192.168.10.12:6789/0, ceph-mon-03-192.168.10.13:6789/0}, election epoch 6, quorum 0,1,2 ceph-mon-01, ceph-mon-02, ceph- mon-03 osdmap e17: 4 osds: 4 up, 4 in pgmap v30: 192 pgs, 3 pools, 0 bytes data, 0 objects
• Page 204Run the following commands on any Proxmox node: # mkdir /etc/pve/priv/ceph
• High Availability Storage for High Availability ClusterIf all went successful thus far, the newly created storage should look like the following screenshot from the Proxmox GUI: PROXMOX Pro Vitual Enviorment
• Chapter 7Preparing a Proxmox node for Ceph Since we are installing Ceph on the same Proxmox node, we are going to set up the network interfaces for a separate network for Ceph traffic only. We will set up three Proxmox nodes-pmxvm01, pmxvm02, and pmxvm03-with Ceph. On all three nodes, we are going to add the following interfaces section in /etc/network/ interfaces. You can also use any IP address that suits your network environment. Run the following command from the Proxmox node pmxvm01:
• High Availability Storage for High Availability ClusterInstalling Ceph Proxmox added a small command-line utility called pveceph to perform various Ceph-related tasks. The following table lists the commands and the tasks that can be performed using pveceph:
• Page 208Please note that the latest version of Ceph is codenamed Firefly, released with major changes such as the omission of Journal for OSD. You should wait several months before using Firefly in the production environment so that known or unknown bugs can be sorted out. There isn't enough usage data yet to be sure about the stability of Ceph's Firefly. If you intend to use Firefly, simply change the version name from emperor to firefly. Ceph's Emperor was the final stable release before Firefly. Run the following commands: root@pmxvm01: /%23 pve ceph install -version firefly root@pmxvm02:/# pveceph install -version firefly root@pmxvm02: /%23 pve ceph install -version firefly
• High Availability Storage for High Availability ClusterClick on Create to open the Monitor creation dialog box, shown in the following screenshot. Select a Proxmox node from the drop-down list and click on the Create button to initiate the Monitor creation. Create: Ceph Monitor
• Chapter 7Creating a new Ceph pool using the Proxmox GUI To create a new pool using the Proxmox GUI, go to Datacenter | <node> | Ceph | Pools. The pool interface shows information about existing pools, such as name, replica number, PG number, and per pool percentage used, as shown in the following screenshot:
• High Availability Storage for High Availability ClusterCreating a Ceph FS Ceph FS requires MDS to function. We have already put together two physical nodes to set up MDS. We are now going to set up an MDS daemon on them to set up Ceph FS. Note that we do not recommend Ceph FS in a production environment yet to store mission-critical virtual machines or other data.
• Chapter 7Setting up Ceph FS using FUSE Unlike the RBD storage, Ceph FS works with Proxmox with mount points. That is, Ceph FS needs to be locally mounted on the Proxmox nodes, and then attach the storage as a local directory to be used with Proxmox. We have to mount the Ceph FS on a directory under /mnt and then connect it to Proxmox through the local directory. Let's create a directory on Proxmox to mount Ceph FS using the following command:
• Page 213etc/fstab: id=admin
• Chapter 7We can store any files, including ISO, templates, backups, and OpenVZ containers, in this storage. Since both RBD and Ceph FS are on the same hardware, we do not recommend that you store backups primarily on Ceph FS. Backup should always be done on separate hardware. The best course of action would be setting up a separate Ceph cluster with Ceph FS only and storing Proxmox backups there. However, for a light load, such as storing ISOs and templates, Ceph FS works great. This eliminates the need to have separate file storage such as FreeNAS or any other type of storage to store ISO images, templates, or OpenVZ containers. Ceph FS has not yet reached industry quality according to the Ceph developers. Minor occasional glitches are to be expected. But from my personal experience, even after using Ceph FS for 11 months straight on a regular basis, I have not came across any glitches or disappointment.
• High Availability Storage for High Availability ClusterExtracting the CRUSH map Before any sort of CRUSH map manipulation, the map has to be extracted from the Ceph cluster. Simply run the following command to extract the CRUSH map:
• Page 216type 0 osd type 1 host type 2 rack type 3 row type 4 room type 5 datacenter type 6 root # buckets host ceph-osd-01 { id -2 # do not change unnecessarily
• Page 217High Availability Storage for High Availability Cluster rule metadata { ruleset 1 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit rule rbd { ruleset 2 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit # end crush map
• Page 218type 2 rack type 3 row typw 4 pdu type 5 room type 6 datacenter type 7 root Chapter 7
• High Availability Storage for High Availability ClusterAdd the following host bucket under the host segment: host ceph-ssd-01 { id -6 alg straw hash 0 host ceph-ssd-02 { id -7 alg straw hash 0
• id -3 # do not change unnecessarily# weight 0.140 alg straw hash 0 # rjenkins1 item osd.1 weight 0.070 item osd.3 weight 0.070 root default { id -1 # do not change unnecessarily #weight 0.280 alg straw hash 0 # rjenkins1 item ceph-osd-01 weight 0.140 item ceph-osd-02 weight 0.140 host ceph-ssd-01 { id -6 # do not change unnecessarily #weight 0.000 alg straw hash 0 # rjenkinsl host ceph-ssd-02 { id -7 # do not change unnecessarily
• Page 221High Availability Storage for High Availability Cluster step emit rule metadata { ruleset 1 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit rule rbd { ruleset 2. type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit rule ssd { ruleset 3 type replicated min_size 1 max_size 10 step take ssd step chooseleaf firstn 0 type host step emit
• Page 222Injecting the CRUSH map into the cluster To inject the new CRUSH map, run the following command: # ceph osd setcrushmap -i crushmapl
• Page 223High Availability Storage for High Availability Cluster device 2 osd.2 device 3 osd. 3 # types type 0 osd type 1 host type 2 rack type 3 row type 4 room type 5 datacenter type 6 root
• host ceph-ssd-02 { id -7 # do not change unnecessarily#weight 0.000 alg straw hash 0 # rjenkinsl root ssd { id -5 # do not change unnecessarily #weight 0.000 alg straw hash 0 # rjenkinsl item ceph-ssd-01 weight 0.000 item ceph-ssd-02 weight 0.000
• Page 225High Availability Storage for High Availability Cluster rule ssd { ruleset 3 type replicated min_size 1 max_size 10 step take ssd step chooseleaf firstn 0 type host step emit # end crush map
• data metadata rbd ssdThe command should show the following result: Chapter 7
• High Availability Storage for High Availability ClusterIt is best to set a cluster to not bring OSDs online right away to prevent rebalancing. Run the following commands to prevent the OSDs from coming online as soon as they are added: #ceph osd set noin
• Page 2280.01999 6
• High Availability Storage for High Availability ClusterWe can see the following output after the command is executed: # id type name up/down reweight weight 0.07959
• Chapter 7Connecting Proxmox to the new pool After we assign the ruleset, our pool is ready to be connected to Proxmox. Perform the following steps mentioned in the Connecting Proxmox to a Ceph cluster subsection in the Installing Ceph using an OS section. We will name the storage cephssd01. So we need to copy the Ceph admin keyring to /etc/pve/priv/ceph/cephssd01.keyring. Then use the information shown in the following screenshot to connect the pool to Proxmox:
• High Availability Storage for High Availability ClusterPlease note that both storage pools will show the exact same size and usage information because they are on the same cluster. The only way to fully separate them is to create two separate Ceph clusters. Clusters can be on the same or different hardware. Ceph benchmarking
• Page 2324. Clear cache again: 3. Test data with block size 4096 bytes for read performance:
• High Availability Storage for High Availability ClusterThe following table shows benchmark results with 26 desktop HDDs as the OSD: Blocksize (bytes) Throughput (MBps)
• Page 234Command ceph osd pool create <name> <pg> <pgs> ceph osd pool delete <name> [<name> --yes-i-really-really- mean-it] ceph osd pool get <name> pg_num ceph osd pool set <name> crush_ ruleset <value> ceph osd get crushmap -o <name> crushtool -d <name> -i <name.txt> crushtool -c <name.txt> -o <name> ceph osd set crushmap -i <name> ceph-fuse -k <keyring> -m <mon:port>/<folder> -o nonempty Fusermount -u /<folder>
• High Availability Storage for High Availability ClusterTotal PG for each pool is 1000. Each replica of pools will have 32 PGs per OSD. Total PGs per OSD for all pools and replicas is 96. The calculations are as follows: Total PGS=(120x100) / 3 replicas = 4000
• Proxmox ProductionLevel Setup 8
• Proxmox Production Level SetupDefining a production level Production level or mode is a stage where a company's cluster environment is fully functional and actively serving its users or clients on a regular basis. It is no longer considered as a platform to learn Proxmox or a test platform to test different things on. A production level setup requires much advanced planning and preparation. This is because once the setup is complete and the cluster has been brought online, it just cannot be taken offline completely at a moment's notice, especially when users become dependent on it. A properly planned production level setup can save hours or days of headache. In this section, we will see how to preplan and choose the right equipment for a production level cluster.
• Current load versus future growthChapter 8 Never design a cluster with only the present day in mind; always look into the future, at least the near future. An enterprise cluster must be able to grow with the company and adapt to the increased operations. However, for the most part, plan in such a way that a cluster does not max out in a very short period of time. Both Proxmox and the Ceph cluster have the ability to grow at any time to any size: this provides the ability to simply add new hardware nodes to expand the size of the cluster and increase the resources required by virtual machines.
• Proxmox Production Level SetupTracking the hardware inventory There are several key pieces of information for the hardware that is being used in a network that an administrator should any time have access to: these include information such as the brand, model and serial number of a hardware component, when was it purchased, who was the vendor, when is it due for replacement, and so on. A proper tracking system can save a lot of time when any of this information needs to be retrieved. Each company is different; accordingly, the tracking system could be different. Nevertheless, the responsibility of gathering this information solely falls on the network manager or administrator. If there is no system in place, then merely creating a simple spreadsheet can be enough to keep track of all hardware-related information.
• Chapter 8The following are examples of desktop- and server-class components from two live clusters currently in service. Both of these examples belong to two low IT budget organizations. Please note that these components are listed only to provide you with a guideline. An i7-based Proxmox node
• Proxmox Production Level SetupBased on this Intel i7 CPU and MSI motherboard, each Proxmox node can have a total of eight cores and 128 GB of memory without spending significant amount of money on server-class hardware. A Xeon-based Proxmox node
• Chapter 8Based on this Intel Xeon CPU and server motherboard, each Proxmox node can have a total of eight cores and 32 GB of memory. Although both the setups are of the similar price range, with the i7 setup we actually stand to gain much more. The performance of i7-4820K is slightly better than Xeon's E3-1245v3, and it provides us with the ability to have a full array of 128 GB of memory for very high-performing virtual machines.
• Proxmox Production Level SetupRAID controller: Intel RS2WC040 6G SAS PCI-E. http://ark.intel.com/products/43165/Intel-RAID-Controller-RS2WC040
• Chapter 8With the entire Hot Swap bay filled with 2 TB HDD, this setup can handle 24 TB of storage without breaking a sweat. If you use 4 TB HDD across all the bays, then install an additional 8 GB of RAM to handle a total storage of 48 TB. When Ceph is set up with three identical nodes with the previously mentioned hardware components, it can provide a very stable shared storage system for the Proxmox cluster. An advanced-level Proxmox production setup When setting up advanced-level enterprise class nodes, quality, stability, and performance usually come before budget. However, the selection has to start from somewhere, and the following setup should be that starting point. There is no place for desktop-class hardware at this stage. The following components or similar should be the minimum standard for an advanced-level setup. Use this information only as guidelines.
• Proxmox Production Level SetupChassis: In-Win IW-RS104-02 Redundant PSU. http://www.in-win.com.tw/Server/zh/goods.php?act=view&id=IW-RS104-02
• Chapter 8• SSD for OS: Intel S3500 SSD 120 GB. ⚫ HDD for storage: Seagate Constellation 2 TB Enterprise.
• Proxmox Production Level SetupHardware tracking No matter which tracking system is used, the goal of hardware tracking is to get information about the components used throughout the Proxmox and Ceph cluster environment. Information such as the serial number, brand/model, due date of components' replacement, and so on must be close on hand of any network manager or administrator. Some companies have a tracking system developed in house, some use a professional inventory tracking system, and some use a plain simple spreadsheet to keep track of the installed components. Although a spreadsheet is a quick and easy way to keep component information handy, it can quickly get out of hand in a cluster environment with several dozen nodes. Have a look at the following excerpt of a simple spreadsheet used by an administrator in his or her production environment:
• Chapter 8The first portion of the spreadsheet has information of all the nodes that include Proxmox, Backup, and Ceph nodes. The second portion of the spreadsheet has information of all the hard drives installed into each 12-bay Hot Swap Ceph node. We can see that eight bays out of 12 have hard drives with the OSD number and the serial numbers related to each hard drive. If we need to find a particular serial number, all we have to do is open the Find option and search for the serial number related to a component. There are also open source options for inventory tracking such as vTiger and Spiceworks. vTiger and Spiceworks both have an asset-tracking option. For a robust network environment with several dozens of nodes and a couple of hundred SSDs or HDDs, the only wise option is to get a real asset-tracking system such as WASP Asset Tracking Software from Wasp Technologies (http://www.waspbarcode.com/ asset-tracking). The paid version of a tracking program has a lot more polished features and is able to handle hardware in different categories and levels.
• Proxmox Production Level SetupMotherboard: Gigabyte GA-990FXA-UD5. The maximum amount of memory supported is 32 GB. http://www.gigabyte.com/products/product-page.aspx?pid=3891#ov
• Chapter 8An AMD-based Ceph setup Since a Ceph node uses a RAID controller and expander card, using an AMD-based component is not recommended. On numerous occasions, we have noticed there is an incompatibility issue with the RAID controller card, AMD chipset, and Ceph. Issues such as not recognizing a RAID card are very comm
• Proxmox Production Level SetupSummary There are no one-setup-fits-all solutions when it comes to a server node setup. The components described in this chapter are mere examples of the possible setups. However, this should provide you with a baseline to direct your thinking when choosing components for the cluster environment you are responsible for. We hope this will aid you in your quest to find that perfect balance between performance and budget that all network administrators crave for.
• Proxmox Troubleshooting9 In this chapter, we are going to learn about some real-life issues taken from Proxmox's production environment and their solutions. Once a Proxmox cluster is set up, it usually runs without issues. However, when issues arise, a system administrator's knowledge is tested. Learning about troubleshooting is like learning from other people's mistakes. Throughout this chapter, we will gain some insight into Proxmox troubleshooting and hope that when we face these in our own clusters, we will know what do at a moment's notice.
• Proxmox TroubleshootingMain cluster issues This section contains issues related to the main Proxmox's cluster operations.
• Chapter 9Rejoining a Proxmox node with the same IP address If you are rejoining a Proxmox node back to the cluster with the same IP address, then the joining command must run with the --force option. Run the following command from the node that is being rejoined:
• Proxmox TroubleshootingThe occurrence of kernel panic when disconnecting USB devices On rare occasions, kernel panic occurs in Proxmox when disconnecting USB devices such as keyboard, mouse, or UPS. There is no real solution to this issue yet as the issue is not reproducible all the time. This issue has been seen on variety of hardware with both standard and nonstandard Proxmox installations. However, almost all the time, the issue does not cause the server to freeze permanently. In almost all cases, the panic can be just ignored and you can go on as usual.
• Chapter 9VMs not booting after you restart the network service On a very rare occasion, virtual machines do not power up after you restart the network service on a Proxmox node. This occurs when corosync and rgmanager services are not automatically loaded when you reboot the Proxmox node. Starting these services will allow the VMs to start again. Use the following commands to start corosync and rgmanager:
• Proxmox Troubleshooting errorProxmox boot failure due to the getpwnam Boot the Proxmox node in recovery mode using the Proxmox installation disk, or select the recovery option from Proxmox's boot menu at the beginning of the boot process. Recovery usually happens on the second line of Proxmox's boot menu, as shown in the following screenshot:
• Chapter 9Booting with a USB stick fails in Proxmox The recommended storage to install Proxmox is SSD or HDD. The only way to install it on a USB stick is to install Debian Linux first and then install Proxmox on top of it by adding the Proxmox VE repositories and installing packages.
• Proxmox TroubleshootingNodes not visible on the Proxmox GUI after an upgrade Proxmox nodes may not be visible after you upgrade to Proxmox VE 3.2. It is a common issue when accessing the Proxmox GUI with the Internet Explorer 11 browser. Use Firefox or the Chrome browser to correctly view the Proxmox GUI. There is a bug report filed for this known issue that is available at https://bugzilla.proxmox.com/ show_bug.cgi?id=475.
• SSH access is possible but Proxmox node does not rebootChapter 9 On numerous occasions, a Proxmox node may not reboot when the process of rebooting is initiated through GUI or from the command prompt. In such cases, the only option available is to physically power cycle a node. In almost all the cases, the node can still be accessed through SSH. If the node is on PDU, it can be forcefully power cycled remotely. However, for those nodes that are not on PDU and physical access is not possible right away, there are the following two methods that you can use to force the process of rebooting the node.
• Proxmox TroubleshootingProxmox cannot mount NFS share due to time-out error Some NFS servers such as FreeNAS do a reverse lookup for hostnames. We have to add Proxmox hostnames to the host files of the NFS server. Use any editor of your choice to open the hosts file:
• Chapter 9Cannot read an iSCSI target even after it has been deleted from Proxmox storage When trying to read the same iSCSI target after it had been deleted from Proxmox storage, an error occurs mentioning the target that has already been added to Proxmox. In these cases, the iSCSI daemon has to be restarted to clear this issue. Run the following command from all the Proxmox nodes. Beware that this command will disconnect all the existing iSCSI targets:
• Proxmox Troubleshooting blocks for CephThe fstrim command does not trim unused To properly trim unused blocks for virtual disks stored on the Ceph storage, perform the following steps:
• CD/DVD Drive (ide2)Network Device (net0) Network Device (net1)
• Proxmox TroubleshootingVM cloning does not parse in Ceph storage When full cloning is performed on a virtual machine stored on Ceph storage, the virtual disk looses parsing. This means that when parsing for VM cloning, Proxmox uses the qemu-1mg method instead of rbd flattening. Until flattening is implemented in later versions of Proxmox, VM clones will lose parsing when stored on Ceph storage.
• Network performance is slower with e1000 VNICChapter 9 The performance of e1000 virtual network interfaces is about 30-35 percent lesser as compared to virtio virtual network interfaces. Changing vNICs to virtio will increase the overall network bandwidth of a virtual machine. The virtio drivers are included in all major Linux flavors. For Windows machines, an ISO file with virtio drivers can be downloaded from http://www.linux-kvm.org/page/WindowsGuestDrivers/
• Proxmox TroubleshootingThe Proxmox 3.2 upgrade adds two com ports and one parallel port to the Windows VM After you upgrade a node to Proxmox VE 3.2, all Windows-based VMs add two com ports and one parallel port as shown in the following screenshot:
• Chapter 9The .vmdk3 format is only supported in pve-qemu-kvm 2.0. Enter the following command to check the version installed in the Proxmox node: # pveversion -v
• Proxmox TroubleshootingThe virtio virtual disk is not available during the Windows Server installation The virtio drivers are not included in the Windows Server installation. During the installation, the Windows setup will not see any virtio virtual disks attached with the virtual machine as shown in the following screenshot:
• Windows SetupSelect the driver to install Red Hat VirtIO Ethernet Adapter (D:\WIN8\AMD64\NETKVM.INF) Red Hat VirtIO SCSI controller (D:\WIN8\AMD64\VIOSTOR.INF) Red Hat VirtiO SCSI controller (D:\WIN8\AMD64\VIOSTOR.INF) Red Hat VirtlO SCSI controller (D:\WIN8\AMD64\VIOSTOR.INF) Red Hat VirtIO SCSI pass-through controller (D:\WIN8\AMD64\VIOSCSI.INF) Red Hat VirtIO SCSI pass-through controller (D:\WIN8\AMD64\VIOSCSI.INF) Red Hat VirtIO SCSI pass-through controller (D:\WIN8\AMD64\VIOSCSI.INF) VirtlO Balloon Driver (D:\WIN8\AMD64\BALLOON.INF)
• Proxmox TroubleshootingOpenVZ containers are no longer shown after a cluster is created This is a common case when a single node Proxmox was set up with an OpenVZ container, and then later when the actual cluster was created and the node in question was added to the cluster. Very likely, all the CT configuration files are in /var/lib/ vz/private. Simply copying the CT configuration files from /var/lib/vz/private to /etc/pve/openvz will make them visible from the Proxmox GUI. The directory location /etc/pve/ is synced over multiple nodes in the cluster to ensure all the nodes have identical configuration. Refer to The Proxmox cluster directory structure section in Chapter 2, Proxmox Under the Hood, for explanation on /etc/pve.
• Backup/restore issuesThis section contains issues related to backing up and restoring Proxmox. Chapter 9
• Proxmox TroubleshootingA backup task takes a very long time to complete, or it crashes when multiple nodes are backing up to the same backup storage When multiple Proxmox nodes are backing up to same the backup storage simultaneously, it tends to take a very long time or the backup is crashed. This is a common occurrence when backup traffic coexists with the main cluster traffic on a gigabit network and the backup node only has one network interface. By separating backups in multiple subnets with multiple network interface cards, we can prevent this issue. Linking the aggregation of multiple network interfaces to form a bigger bandwidth, as shown in the following screenshot, can also help to avoid backup traffic congestion. Proxmox 1
• Page 274107: INFO: status: 80% (2706263244/4294967296), sparse 16% (698703462), duration 1950, 5/4 MB/s 107: ERROR: interrupted by signal
• Proxmox TroubleshootingThe SPICE console has become unstable after the Proxmox VE 3.2 update After clicking on the SPICE button for a SPICE-enabled VM, the browser gives an empty ticket error. This is a common occurrence if the browser cache is not updated after a Proxmox upgrade. Just refreshing the browser will fix this issue.
• Chapter 9New Inbound Rule Wizard Protocol and Ports
• Page 277
• Page 27810 Putting It All Together
• Putting It All TogetherThe Ceph cluster has been used as a primary storage backend in all these scenarios. The latest Proxmox VE Version 3.2 supports co-existence of Proxmox and Ceph on the same node. This is a Proxmox technology preview in this version. This preview is solely the inclusion of Ceph in the Proxmox hypervisor and not the Ceph itself. Ceph is very much well established and stable. For the purpose of stability and any unknown bugs, we are going to use a separate Ceph cluster in network diagrams. All IP addresses and hostnames used in this chapter are fictitious. Any resemblances to real networks or hostnames are purely coincidental.
• DescriptionSubnet This is the main administrative and professors' network. Private Wi-Fi for professors' usage is an extension of this network to allow professors to retrieve their files in a wireless fashion. All classrooms are also on this network to provide in-class access of files for professors. This is the storage cluster.
• Putting It All TogetherIf you are going to use the latest Proxmox VE 3.2 to create a multitier Ceph cluster, keep in mind that as of now, the Proxmox GUI does not support multi-Ceph clusters. You can only manage one cluster through the Proxmox GUI and additional Ceph clusters will have to be managed through the CLI. Multitiered infrastructure is very typical for data centers, where there is a different level of SLA-based clients with various requirements for storage performance.
• Virtual bridge vmbr10 vmbr20 vmbr30 vmbr40Description vmbr50 vmbr60 This is the bridge for the subnet 192.10.10.0 of the company ABC This is the bridge for the subnet 192.20.20 of the company XYZ.0 This is the bridge for the OpenVZ containers for web hosting instances This is the bridge for the Object Storage instances to be used by a software developer
• Putting It All TogetherFour subnets used in this example are listed in the following table: Subnet
• Chapter 10The network 192.200.200.0 is controlled, managed, and isolated using a virtual firewall VM5. The virtual firewall has two vNICS, one for a WAN connection through vmbr3, and the other to connect to the dedicated NIC on the Proxmox node through vmbr4. The eth2 network interface card of Proxmox node is connected to a separate LAN switch to only connect public devices. The virtual firewall provides the ability to monitor Internet traffic to keep in line with any violation of the library's Internet usage policy. Each Proxmox nodes has four network interface cards, eth0, eth1, eth2, and eth3, and the cluster has three virtual bridges, vmbr0, vmbr2, and vmbr4. The main storage cluster is connected to the Proxmox node through eth1 and the backup cluster is connected to eth3.
• Putting It All TogetherScenario #7 - virtual infrastructure for hotel industry The key requirements for this scenario are as follows:
• Page 286Chapter 10 In this scenario, a geographical survey company has a main office and a branch office connected with a 1+GBps hardlink network connectivity. Each office has an identical infrastructure setup. All surveyors use mobile devices such as tablets for their survey work. The survey software autodetects which office IP is live and sends data to the infrastructure of that office. All data is replicated on a block level in real-time between two offices. If the infrastructure of one office becomes unavailable, the staff can simply continue to work out of infrastructure from the other office.
• Putting It All TogetherThe following is the network diagram for Scenario #2 - multitier storage cluster with Proxmox cluster: SCENARIO #2
• Chapter 10The following is the network diagram for Scenario #3 - virtual infrastructure for multitenant cloud service provider: Small Software Company Company Company Business 2 Business 1 Developer 123
• Putting It All TogetherThe following is the network diagram for Scenario #4 - a nested virtual environment for a software development company: SCENARIO #4
• Chapter 10The following is the network diagram for Scenario #5 - a virtual infrastructure for the public library: SCENARIO #5
• Putting It All TogetherThe following is the network diagram for Scenario #6 - multifloor office virtual infrastructure with virtual desktops: SCENARIO #6
• Chapter 10The following is the network diagram for Scenario #7 - virtual infrastructure for hotel industry: SCENARIO #7
• Putting It All TogetherThe following is the network diagram for Scenario #8, virtual infrastructure for a geological survey organization: SCENARIO #8
• SummaryChapter 10 We do hope that these network diagrams were helpful to understand the flexibility and customizability of Proxmox. These are just very few scenarios out of hundreds that you might face in the real world. The concept of network diagraming will also help you visualize a network before you start implementing it. A network diagram on paper helps you to figure out data path or connectivity before implementing or documenting an existing network to see how the network environment is set up. Although we are at the end of this book, your journey into the world of virtualization using Proxmox is just beginning. If the knowledge from this book has helped you to understand Proxmox better and you actually have leaped forward many steps, then this book has fulfilled its purpose. I would personally like to thank you for going through the journey together and wish you all the best in your virtualization career.
• Page 295
• Symbols192.110.10.0 subnet 259 192.160.10.0 subnet 259
• Page 297APC-managed PDU user creating 147, 148 args: option 57 autostart: option 57 B backup/restore issues 251, 252
• Page 298ceph-deploy install <node> command 212 ceph-deploy mds create <node> command 212 ceph-deploy mds destroy <node> command 212 ceph-deploy mon create <node> command 212 ceph-deploy mon destroy <node> command 212 ceph-deploy osd create <node>:/dev/sdX command 212 ceph-deploy tool installing 176, 177 Ceph FS about 166 creating 190
• Page 299cloud computing URL 137 cluster logfile 65 cluster map 163 cluster operation, issues fencing, disabling 233 GRUB endless loop error 238 kernel panic 234 node invisibility 238 node rebooting error 239 offline status, in GUI 232 out of Quorum error 235
• entry-level Proxmox production setup about 218, 21917-based Proxmox node 219, 220 Xeon-based Proxmox node 220, 221 external network virtualization 114 F
• Intel CPU performance versus AMD CPU performance 229Intelligent Platform Management Interface. See IPMI internal network virtualization 114
• N name: option 58 NAS4FreeURL 86 NAT about 119 adding 130 nested virtual environment about 93, 94 enabling 95
• Page 303о Object Storage Daemon. See OSD
• Page 304Proxmox hypervisor 6 Proxmox node preparing, for Ceph 185 Proxmox node #1 (pmxvm01) 28 Proxmox VE installing, on Proxmox nodes 27, 28 Proxmox Wiki S sata(n): option 59 scenarios academic institution 258 multifloor office virtual infrastructure with virtual desktops 263 multitier storage cluster, using Proxmox cluster 259 nested virtual environment, for software development company 261 network diagrams 265, 266 virtual infrastructure, for geological survey about 245
• Page 305startdate: option 59 startup: option 60 STONITH 146 storage system issues Ceph FS, mounting 243 damaged LVM storage 239 iSCSI target reading error 241 mode session exit code 21 errors 240
• Page 306virtual disks image .qcow2 image type 75, 76 .raw image type 76, 77 .vmdk image type 77 about 74 file, manipulating 77 supported image formats 74 virtual image file manipulating 77 moving 79,80 resizing 78, 79 virtual infrastructure, for geological survey organization key requirements 264 network diagram 272 virtual infrastructure, for hotel industry about 264 key requirements 264 network diagram 271 virtual infrastructure, for multitenant cloud service provider key requirements 260 network diagram 267 vmbr0 260 vmbr5 260 vmbr10 261 vmbr20 261 vmbr30 261 vmbr40 261 vmbr50 261 vmbr60 261 virtual infrastructure, for public library 192.160.10.0 subnet 262
• Page 307vmbr1, virtual bridge 260 vmbr5, virtual bridge 260 vmbr10, virtual bridge 261 vmbr20, virtual bridge 261 vmbr30, virtual bridge 261 vmbr40, virtual bridge 261 vmbr50, virtual bridge 261 vmbr60, virtual bridge 261 vmbrX 118 VNC/SPICE console issues 253,254 vNIC 15, 117 vzdump.conf file about 107
• [PACKT] open source community experience distilledPUBLISHING Thank you for buying Mastering Proxmox
• [PACKT open source community experience distilledPUBLISHING Microsoft System Center Virtual Machine Manager 2012 Cookbook Paperback: 342 pages
• [PACKT open source community experience distiledPUBLISHING Getting Started with Citrix XenApp 6