Search

Docker Vs Virtual Machines(VMs)

Let’s have a quick warm up on the resource management before we dive into the discussion on virtualization and dockers.In today’s multi-technology environments, it becomes inevitable to work on different software and hardware platforms simultaneously.The need to run multiple different machines (Desktops, Laptops, handhelds, and Servers) platforms with customized hardware and software requirements has given the rise to a new world of virtualization in IT industry.What a machine need?Each computing environment(machine) needs its own component of hardware resources and software resources.As more and more machines are needed, building up and administering many such stand-alone machines is not only cumbersome, time consuming but also adds up to the cost and energy.Apparently; to run a customized High-power Scalable Server is a better idea to consolidate all the hardware and software requirements into one place and have a single server run and distribute the resources to many machines over a network.That saves us time, resources, energy and revenue.These gigantic servers are stored in a data warehouse called a Datacenter.Below Diagram (2) indicates a single server serving and sharing resources and data among multiple client machinesDoes this look simplified enough? Yes of course!So, this setup looks feasible we have a high-power, high-storage Server that gives resources to many smaller(resources) machines over a network.How to manage huge data - ServersWith Internet Of Things in boom, Information is overflowing with a huge amount of data; handling tremendous data needs more system resources which means more Dedicated servers are needed.Many Servers approach challenge:Running several Dedicated servers for specific services such as Web service, application or database service as indicated in Diagram (3) is difficult to administer and consumes more energy, resources, manpower and is highly expensive.In addition; resource utilization of servers is very poor resulting in resource wastage.This is where simulating different environments and running them all on a single server is a smart choice; rather than having to run multiple physically distinct servers.This is how Diagram (3) would change after consolidating different servers into one as shown in Diagram (4).Sheet 2VirtualizationWhat is VirtualizationThe above single server implementation can be defined as the following term.Virtualization is a technique used to simulate and pretend a single infrastructure resource (hardware resources and software resources) to be acting as many providing multiple functionalities or services without the need to physically build, install and configure.In other words;Running multiple simulated environments in a single machine without installing and configuring them is called Virtualization.Technically speaking;Virtualization is an abstract layer that shares the infrastructure resources among various simulated virtual machines without the need to physically set up these environments.Diagram (5) displays different virtual Operating systems are running on the same machine and using the same hardware architecture of the underlying machine.What is a Virtual machineThe simulated virtualized environments are called virtual machines or VM.Virtual machine is a replication/simulation of an actual physical machine.A VM acts like a real physical machine and uses the physical resources of the underlying host OS.A VM is a running instance of a real physical machine.Need for virtualizationSo; we have an overview of virtualization, let us examine when should we virtualize and what are the benefits of virtualization?Better resource management and cost-effective: as indicated in Diagram (6) and Diagram (7); hardware resources are distributed wisely on need basis to different environments; all the virtual machines share the same resources and reduce resource wastage.Ease of quick administration and maintenance: It is easier to build, install, configure one server rather than multiple servers. Updating a patch on various machines from a single virtualized server is much more feasible.Disaster recovery: Since all the virtualized machines reside on the same server and are treated as mounted volumes of data files, it is easier to back up these machines. In case of a disaster failure (power failure, network down, cyber-attacks, failed test code, etc) VM screenshots are used to recover the running state of the machine and the whole setup can be built up within minutes.Isolated and independent secure test environment: virtualization provide an isolated independent virtual test environment to test the legacy code or a vendor-specific product or even a beta release or say a corrupt code without affecting the main hardware and software platform. (This is a contradictory statement though; will discuss more under types of virtualization)These test environments like dev, uat, preprod, prod etc..can be easily tested and discarded.Easily scalable and upgradable: Building up more simulated environments means spinning up more virtual machines. Also upgrading VMs is as good as to run a patch in all VMs.Portable: Virtual machines are lightweight compared to the actual running physical machines; in addition, a VM that includes its own OS, drivers, and other installation files is portable on any machine. One can access the data virtually from any location.The screenshot of activity monitor below compares the CPU load:Implementation a) What is hypervisor and its types?As discussed in the previous section; virtualization is achieved by means of a virtualized layer on top of hardware or a software resource.This abstract layer is called a hypervisor.A hypervisor is a virtual machine monitor (VMM)There are 2 types of hypervisors: Diagram (8)Type-1 or bare-metal hypervisorType-2 or hosted hypervisorType-1 or bare-metal hypervisor is installed directly on the system hardware, thus abstracting and sharing the hardware components with the VMs.Type-2 or hosted hypervisor is installed on top of the system bootable OS called host OS; this hypervisor abstracts the system resources visible to the host OS and distributes it among the VMs.Both have their own role to play in virtualization.b) Comparing hypervisor typesType-1 or bare-metal hypervisorType-2 or hosted hypervisorInstalled directly on the infrastructure-OS independent and more secure against software issues.Installed on top of the host OS-more prone to software failures.Better resource flexibility: Have direct access to the hardware infrastructure (Hard-drive partition, RAM, embedded cards such as NIC). Provide more flexibility and scalability to the VMs and assign resources on a need basis.Limited resource allocation: Have access to just the resources exposed by the host OS.VMs installed will have limited access to hardware resources allocated and exposed by the host OS.Single point of failure: A compromised VM may affect the kernel. Extra security layers needed.A compromised VM may affect only the host OS, kernel still remains unreachable.Low latency due to direct link to the infrastructure.High latency as all the VMs have to pass through the OS layer to access the system resources.Generally used in ServersGenerally used on small client machinesExpensiveLess expensiveType-1 Hypervisors in market:VMWare ESX/ESXiHyperkit (OSX)Microsoft Hyper-V (Windows)KVM(Linux)Oracle VM ServerType-2 Hypervisors in market:Oracle VM VirtualBoxVMWare WorkstationParallels desktop for MACTypes of virtualizationBased on what resource is virtualized, there are different classifications of virtualization.Server, Storage device, operating system, networkDesktop virtualization: Entire desktop environment is simulated and distributed to run on a single server all at once. A desktop virtualization allows administrators to manage, install, configure similar setups on many machines. Upgrading all the machines with a single patch update or security checks becomes easier and faster.Server virtualization: Many dedicated servers can be virtualized into a single server that provides multi-server functionality.Example:Many virtual machines can be built up sharing the same underlying system resources.Storage, RAM, disks, CPUOperating system virtualization: This happens at the kernel level Hypervisor on hardware type 2 bare-metal One machine: Can boot up as multiple OS like Windows or Linux side-by-sideApplication virtualization: Apps are packaged and stored in a virtual environment and are distributed across different VMs. Example Microsoft applications like excel, MS word, Powerpoint etc, Citrix applications.Network functions virtualization: Physical network components such as NIC cards, switches, routers, servers, hubs, and cables are all assembled in a single server and used virtually by multiple machines without having the load of installing them on every machine.Virtualization is one of the building blocks and driving force behind cloud computing.Cloud computing provide virtualized need-based services. This has given an uplift to the concept of virtualization.A quick mention of various cloud computing models/services are listed below:SaaS – Software as a Service– end-user applications are maintained and run by service providers and easily distributed and used by the end users without having to install them.Top SaaS providers: Microsoft (Office suite, CRM, SQL server databases), AWS, Adobe, Oracle (ERP, CRM, SCM), Cisco’s Webex, GitHub ( git hosting web service)PaaS – Platform as a Service – computing infrastructure(hardware/software) is maintained and updated by the service provider and the user just have to run the product over this platform.Top Paas providers: AWS beanstalk, Oracle Cloud Platform (OCP), Google App EngineIaaS – Infrastructure as a Service – Provide infrastructure such as servers, physical storage, networking, memory devices etc. Users can build their own platform with customized operating system and applications.Key IaaS providers: Amazon Web Services, Microsoft Azure, Google compute engine, CitrixConclusion:We now have a fair understanding of types of virtualization and how they are implemented.ContainerizationThough virtualization has its pros; there are certain downsides of virtualization such as:Not all systems can be virtualized always.A corrupt VM is sometimes contagious and may affect other VMs or the kernel in-case of a Type-1 or bare-metal hypervisor.Latency of virtual disks due to increased payload on the CPU resources with a higher number of VMsUnstable performanceAn alternative approach to overcome the above flaws of virtualization is to Containerize the applications and the run-time environment together.What is containerization  Containerization is an OS-level virtualization; wherein the entire build of an application along with run-time environment is encapsulated or bundled up in a package.These packages are called containers.Containers are lightweight virtualized environments. These are independent of the infrastructure both hardware and software.The run-time environment includes the operating system, binaries, libraries, configuration files and other applications as shown in Diagram (9).What is DockersDockers provide an excellent framework for containerization and allow to build, ship, and run distributed applications over multiple platforms.Docker framework is setup as a docker engine installed on host OS and a docker daemon (background process) process is started that manage the virtual containers.Refer Diagram (10) that shows a Docker engine with 3 containers residing on host OS (MAC OS).An instruction file called dockerfile is written with a set of system commands that change the filesystem such as add, copy or delete commands, run commands, install utilities, system calls etc…This dockerfile is built and packaged along with its run-time environment as an executable file called a docker image.Docker daemon services run these images to create docker containers.Docker container is a run-time instance of an imageIt is wise to say that many images (or layers of instruction files) make up a container.Docker containers have a compact packaging and each container is well isolated.We can run, start, stop, attach, move or delete containers as these runs as services on the host OS.Each image is made up of different layers; each image based on top of the other with the customized command changes that we make.Every time we make a change in the filesystem, each change related to the image is encapsulated in a new layer of filesystem and stacked up above the parent image.Only the changed layers are rebuilt, rest of the unchanged image layers are reused.Certain docker commands ADD, RUN and COPY create a new layer with increased byte size; rest of the commands simply adds up a new layer with zero-byte size.These layers are re-used to build a new image, hence faster and lightweight.Docker images are alsoThe layer approach of an image every time there is a change in the image makes it possible to Version control the docker images.Here is a terminal recording that shows docker engine process and how images and containers are created.Docker documentation - to create containers.Ppt diagram:Code -> package -> build images -> registry hub -> download/pull image -> run containerAnimation: sheet4Let’s consider the docker container: divyabhushan/learn_docker hosted on docker hub.Latest tagged image: centOS_release1.2What is the container environment?Base OS: Centos:7Utilities: vim, yum, gitApps/files: Dockerfile, myApp.sh, runtests.sh, data and other supporting files.Git source code: dockerImagesDownload as: git clone https://github.com/divyabhushan/DockerImages_Ubuntu.gitWhat does the container do?Container launches “myApp.sh” in Ubuntu:14.04 environment and run some scripts along with a set of post test_suites in the container (Ubuntu:14.04) and saves the output log file.How to modify and build your own appStep 1: pull 1.1: Pull the docker image1.2: Run image to create a container and exitStep 2: modify2.1: Start the container2.2: Attach to the container and make some changesStep 3: commit3.1: Examine the history logs and changes in the container3.2: Commit the changes in containerStep 4: push4.1: Push new image to docker hubLet us see the steps in action:Step 1: pull docker image on your machine1.1: Pull the docker imageCommand:docker pull divyabhushan/learn_docker:myApp_ubuntu_14.04View the image on systemdocker imagesscreenshotCommand:docker run -it --name ubuntu14.04 0a6f949131a6Run command in ubuntu container and exit, the container is stopped on exiting out.View the stopped container with the ‘ps -a’ command.Step 2: modifyStart the containerCommand:docker start <container_id>Now the container is listed as a running process Attach to the container and make some changesCommand:docker attach 7d0d0225778cedit the ‘git configuration’ file and ‘myApp.sh’ scriptContainer is modified and stoppedStep 3: commitExamine the history logs and changes in the containerThe changes done inside the container filesystem can be viewed using the ‘docker diff’ command as:Command: docker diff 7d0d0225778cCommit the changes in containerDocker commit:Usage: docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]docker commit -m 'new Ubuntu image' 7d0d0225778c divyabhushan/learn_docker:ubuntu14.04_v2New image is created and listedStep 4: pushPush new image to docker hubCommand:docker push divyabhushan/learn_docker:ubuntu14.04_v2Point to note: just the latest commit change layer ‘50a5ce553bba’ has been pushed, while the other layers were re-used.Image available on docker hub:The latest tagged image can now be pulled from other machines; and run to create the same container environment.Conclusion: An image was pulled and run to create a container to replicate the environment. Container was modified, new changes were committed to form a new image. New Image pushed back on the docker hub and now available as a new tag ready to be pulled by other machines.Difference between Dockers and Virtual machinesTabular differences on various parametersParametersVMsDockersarchitectureHardware level virtualization. Each VM has its own copy of OS.Software level virtualization. Dockers have no own OS, run on host OSIsolationFully isolatedProcess or application-level isolation.  InstallationHypervisor can run directly on the hardware resources or on the host OS.Docker engine is installed on top of the host OS. A docker daemon process is initiated on the host OS. There is no separate OS for every container.CPU processing + performanceSlower: A VM contains the entire run-time environment that has to be loaded every time. Uses more CPU cycles; gives unstable performance.Faster: Docker images are pre-built and share host resources as a result running an image as a container is lightweight and consumes less CPU cycle; gives a stable performanceHardware storageMore storage space as each VM is an independent machine (OS). Example: 3 VMs of 800MB each will take 2.4 GB of space.Docker containers are lightweight since do not require to load OS+drivers, run on host OS as processes.PortableDependency on host OS and hardware makes VM less portable. Importing a VM still requires manual setup such storage, RAM and network.Highly portable since lightweight and zero dependency on hardware.Scalable and code-reusabilitySpinning up more VMs still need administrative tasks such as distributing resources to VM. Running a new machine puts extra load on the system resources also re-managing earlier VMs becomes a task. Every VM keeps its own copy of resources-poor code-reusability.Spinning up new docker containers simply means running pre-built images into containers as a process inside host OS. Containers are also configured on-the-fly passing parameters and run-time. Single image can be run and used to create many containers; encourage code-reusabilityResource utilizationStatic allocation results in resource wastage in case of idle VMs or if a VM’s resource requirement increases.Resources are dynamically allocated and de-allocated on the need basis by the docker engine.Docker system prune or garbage collectionVirtual machines do not have an in-built prune mechanism, these have to be administered manually.Docker image and containers can be pruned; which frees up a sensible amount of storage and memory space and CPU cycles.New environmentCreating new VM from the scratch is a tedious, repetitive tasks. It involves installing a new OS, loading kernel drivers and other tools and configurations.Package the code and dependency files, build into an image, run the image to create a new container. Use an existing or a base image (dockerhub- scratch) to run and create more containers on the go.Web-hosted HubNo web hosted hub for VMsdockerHub provides an open-source reliable trusted source of pre-built images that can be downloaded to run new containers.Version control (backup, restore,track history)(refer git)Snapshot of VMs are not very user-friendly and consume more space.Docker images are version controlled. Every delta difference in each docker container can easily be viewed (demo: docker diff <container_id>). Any change in the image is stored as a different layered version. A reference link to older images saves build time and space.Auto-buildAutomation of creating VMs is not very feasible.Docker images can also be auto-built from every source code check-in to GitHub (Automated builds on Dockerhub)Disaster recoveryTedious to recover from VM backup files.Easier to restore docker images (like files) just like git source files in case images are version controlled. Backup images only have to be run to create containers. (refer: screenshot).UpdateAll the VMs have to updated with the release patch.A single image is updated, re-built and distributed across multiple platforms.Memory usage+speedSlower: Entire snapshot of a machine and the OS is loaded into the cache memory.Real-time and fast: pre-built images. Only the instance, i.e, a container has to be run as a process and uses memory like an executableData integrityVM behavior may change if the dependency includes beyond the VM boundaries. (example: an app depends on production host network settings)Same behavior of apps in any environmentsecurityMore secure: A failure inside a VM may reach its guest OS but not the host OS or other virtual machines. Type-2 hypervisor though has a risk of kernel attack.Less secure: If a docker container compromised; underlying OS and hence all the containers may be affected since they share the same host kernel. OS Kernel may also be risked.Key providersRed hat KVM, VMWare, Oracle VM VirtualBox, Mircrosoft Hyper-V, Citrix XenServerDockers, Google kubernetes Engine, AWS Elastic Container serviceData authenticationLot of software licenses.Docker maintains inbuilt content trust to verify published images. When to use VM or a DockerWhen the need is an isolated OS, go for VMs.For a hardware and software independent isolated application that needs fast distribution on multiple environments, use dockers.Docker use-case:Example: A database application along with its databaseConsider the docker image - Oracle WebLogic Server on Docker Hub.This image is pre-built Oracle WebLogic Server runtime environment, including Oracle Linux 7 and Oracle JDK 8 for deploying Java EE applications.To create Server configurations on any machine, just download this image and run to create and start a container.There is no need to install and configure JDK, Linux or other run-time environment.Do not use Docker use-case:The application depends on utility outside the docker container.Code developed on dev machine with base OS as MAC; needs certain firewall setting on say Ubuntu OS.How can the code be tested on the production ubuntu OS firewall while running from MAC OS docker container?Solution:  Install a virtualization software on host OS-MAC; Create a VM (Virtual machine) with host OS as Ubuntu (same as production environment).Configure the desired firewall settings on host VM – Ubuntu; import the test code inside Ubuntu and test.Use a VM:For Embedded systems programming, a VM is installed that connects to the system device drivers, controllers and kernel.Virtualization used along with docker:An extension to the previous scenario would be if you would want to also test your python application in the host OS-Ubuntu VM without having to set up the python exe and its libraries and binaries.All you have to do is: Install Docker engine for Ubuntu OS and pull the python image from Docker hub as:docker pull python:tag [ tag is the python version-choose the appropriate version ]docker pull python:2.7Refer: Python imageEither write a Dockerfile to import/copy entire source code to python environment or directly run the image passing the script path as below:Command:$docker run -it --name my-python-script -v “$PWD”:/usr/src/myapp -w /usr/src/myapp python:2.7 python my-application.pyCommand options:-v = volume list-bind mount a volume [mount present working directory onto /usr/src/myapp inside container]-w = workdir string-working directory inside the containerMoreover; you can also test your python code in more than one version by downloading different python images, running them to create different containers and running your app in each container.What’s exciting here is that once the code tested in each python environment; you could quickly work on the test results and drop the containers. And deploy the code to production only once code tested against various python versions.Final thoughtsVMs and dockers are compatible with each other. Dockers are not here to replace Virtual machines.Both serve the same purpose of virtualizing the computing and infrastructure resources for optimized utilization.Using both Virtual machines and dockers together can yield better results in virtualization.When one desires a fast, lightweight, portable and highly scalable hardware-independent environment for multiple applications isolation; wherein security is not the major concern; Dockers is the best choice.Use a VM for embedded systems that are integrated with hardware; such as device driver or kernel coding.A scenario simulating an infrastructure setup with a high resource control and dependency on system resources; VMs are a better choice.Use of Dockers inside VMCI/CD pipelines scenario:Virtualization enables a smooth CI/CD process flow by promoting the users to concentrate only on developing the code on a working system that is set up for automated continuous integration and deployment without having to duplicate the entire setup each time.A virtualized environment is set up; either using a VM or a docker image that takes care of the automatic code check-ins, builds, regression testing, and deployments on the server.

Docker Vs Virtual Machines(VMs)

8K
Docker Vs Virtual Machines(VMs)

Let’s have a quick warm up on the resource management before we dive into the discussion on virtualization and dockers.

In today’s multi-technology environments, it becomes inevitable to work on different software and hardware platforms simultaneously.

The need to run multiple different machines (Desktops, Laptops, handhelds, and Servers) platforms with customized hardware and software requirements has given the rise to a new world of virtualization in IT industry.

What a machine need?

Each computing environment(machine) needs its own component of hardware resources and software resources.

As more and more machines are needed, building up and administering many such stand-alone machines is not only cumbersome, time consuming but also adds up to the cost and energy.

Apparently; to run a customized High-power Scalable Server is a better idea to consolidate all the hardware and software requirements into one place and have a single server run and distribute the resources to many machines over a network.

That saves us time, resources, energy and revenue.

A Server with many hardware components installed in a datacenter

These gigantic servers are stored in a data warehouse called a Datacenter.

Below Diagram (2) indicates a single server serving and sharing resources and data among multiple client machines

Single server sharing data with many machines

Does this look simplified enough? Yes of course!

So, this setup looks feasible we have a high-power, high-storage Server that gives resources to many smaller(resources) machines over a network.

How to manage huge data - Servers

With Internet Of Things in boom, Information is overflowing with a huge amount of data; handling tremendous data needs more system resources which means more Dedicated servers are needed.

Many servers for different computing needs

Many Servers approach challenge:

Running several Dedicated servers for specific services such as Web service, application or database service as indicated in Diagram (3) is difficult to administer and consumes more energy, resources, manpower and is highly expensive.

In addition; resource utilization of servers is very poor resulting in resource wastage.

This is where simulating different environments and running them all on a single server is a smart choice; rather than having to run multiple physically distinct servers.

This is how Diagram (3) would change after consolidating different servers into one as shown in Diagram (4).

Sheet 2

Servers after virtualization

Servers after virtualization

Virtualization

What is Virtualization

The above single server implementation can be defined as the following term.

Virtualization is a technique used to simulate and pretend a single infrastructure resource (hardware resources and software resources) to be acting as many providing multiple functionalities or services without the need to physically build, install and configure.

In other words;

Running multiple simulated environments in a single machine without installing and configuring them is called Virtualization.

Technically speaking;

Virtualization is an abstract layer that shares the infrastructure resources among various simulated virtual machines without the need to physically set up these environments.

A single machine running multiple operating systems

Diagram (5) displays different virtual Operating systems are running on the same machine and using the same hardware architecture of the underlying machine.

What is a Virtual machine

The simulated virtualized environments are called virtual machines or VM.

Virtual machine is a replication/simulation of an actual physical machine.

A VM acts like a real physical machine and uses the physical resources of the underlying host OS.

A VM is a running instance of a real physical machine.

Need for virtualization

So; we have an overview of virtualization, let us examine when should we virtualize and what are the benefits of virtualization?

  1. Better resource management and cost-effective: as indicated in Diagram (6) and Diagram (7); hardware resources are distributed wisely on need basis to different environments; all the virtual machines share the same resources and reduce resource wastage.
  2. Ease of quick administration and maintenance: It is easier to build, install, configure one server rather than multiple servers. Updating a patch on various machines from a single virtualized server is much more feasible.
  3. Disaster recovery: Since all the virtualized machines reside on the same server and are treated as mounted volumes of data files, it is easier to back up these machines. In case of a disaster failure (power failure, network down, cyber-attacks, failed test code, etc) VM screenshots are used to recover the running state of the machine and the whole setup can be built up within minutes.
  4. Isolated and independent secure test environment: virtualization provide an isolated independent virtual test environment to test the legacy code or a vendor-specific product or even a beta release or say a corrupt code without affecting the main hardware and software platform. (This is a contradictory statement though; will discuss more under types of virtualization)
    These test environments like dev, uat, preprod, prod etc..can be easily tested and discarded.
  5. Easily scalable and upgradable: Building up more simulated environments means spinning up more virtual machines. Also upgrading VMs is as good as to run a patch in all VMs.
  6. Portable: Virtual machines are lightweight compared to the actual running physical machines; in addition, a VM that includes its own OS, drivers, and other installation files is portable on any machine. One can access the data virtually from any location.

Sheet 3 Resource management of resources

The screenshot of activity monitor below compares the CPU load:

Percentage of CPU resources without and with OS virtualization

Implementation 

a) What is hypervisor and its types?

As discussed in the previous section; virtualization is achieved by means of a virtualized layer on top of hardware or a software resource.

This abstract layer is called a hypervisor.

A hypervisor is a virtual machine monitor (VMM)

There are 2 types of hypervisors: Diagram (8)

  1. Type-1 or bare-metal hypervisor
  2. Type-2 or hosted hypervisor

Type-1 or bare-metal hypervisor is installed directly on the system hardware, thus abstracting and sharing the hardware components with the VMs.

Type-2 or hosted hypervisor is installed on top of the system bootable OS called host OS; this hypervisor abstracts the system resources visible to the host OS and distributes it among the VMs.

Both have their own role to play in virtualization.

b) Comparing hypervisor types

Type-1 or bare-metal hypervisorType-2 or hosted hypervisor

Installed directly on the infrastructure-OS independent and more secure against software issues.

Installed on top of the host OS-more prone to software failures.

Better resource flexibility: Have direct access to the hardware infrastructure (Hard-drive partition, RAM, embedded cards such as NIC). Provide more flexibility and scalability to the VMs and assign resources on a need basis.

Limited resource allocation: Have access to just the resources exposed by the host OS.

VMs installed will have limited access to hardware resources allocated and exposed by the host OS.

Single point of failure: A compromised VM may affect the kernel. Extra security layers needed.

A compromised VM may affect only the host OS, kernel still remains unreachable.

Low latency due to direct link to the infrastructure.

High latency as all the VMs have to pass through the OS layer to access the system resources.

Generally used in Servers

Generally used on small client machines

Expensive

Less expensive

Type-1 Hypervisors in market:

VMWare ESX/ESXi

Hyperkit (OSX)

Microsoft Hyper-V (Windows)
KVM(Linux)

Oracle VM Server

Type-2 Hypervisors in market:

Oracle VM VirtualBox

VMWare Workstation

Parallels desktop for MAC

Type-1 and type-2 hypervisor

Types of virtualization

Based on what resource is virtualized, there are different classifications of virtualization.

Server, Storage device, operating system, network

Desktop virtualization: Entire desktop environment is simulated and distributed to run on a single server all at once. A desktop virtualization allows administrators to manage, install, configure similar setups on many machines. Upgrading all the machines with a single patch update or security checks becomes easier and faster.

Server virtualization: Many dedicated servers can be virtualized into a single server that provides multi-server functionality.

Example:

Many virtual machines can be built up sharing the same underlying system resources.

Storage, RAM, disks, CPU

Operating system virtualization: This happens at the kernel level Hypervisor on hardware type 2 bare-metal One machine: Can boot up as multiple OS like Windows or Linux side-by-side

Application virtualization: Apps are packaged and stored in a virtual environment and are distributed across different VMs. Example Microsoft applications like excel, MS word, Powerpoint etc, Citrix applications.

Network functions virtualization: Physical network components such as NIC cards, switches, routers, servers, hubs, and cables are all assembled in a single server and used virtually by multiple machines without having the load of installing them on every machine.

Virtualization is one of the building blocks and driving force behind cloud computing.

Cloud computing provide virtualized need-based services. This has given an uplift to the concept of virtualization.

A quick mention of various cloud computing models/services are listed below:

SaaS – Software as a Service– end-user applications are maintained and run by service providers and easily distributed and used by the end users without having to install them.

Top SaaS providers: Microsoft (Office suite, CRM, SQL server databases), AWS, Adobe, Oracle (ERP, CRM, SCM), Cisco’s Webex, GitHub ( git hosting web service)

PaaS – Platform as a Service – computing infrastructure(hardware/software) is maintained and updated by the service provider and the user just have to run the product over this platform.

Top Paas providers: AWS beanstalk, Oracle Cloud Platform (OCP), Google App Engine

IaaS – Infrastructure as a Service – Provide infrastructure such as servers, physical storage, networking, memory devices etc. Users can build their own platform with customized operating system and applications.

Key IaaS providers: Amazon Web Services, Microsoft Azure, Google compute engine, Citrix

Conclusion:

We now have a fair understanding of types of virtualization and how they are implemented.

Containerization

Though virtualization has its pros; there are certain downsides of virtualization such as:

  • Not all systems can be virtualized always.
  • A corrupt VM is sometimes contagious and may affect other VMs or the kernel in-case of a Type-1 or bare-metal hypervisor.
  • Latency of virtual disks due to increased payload on the CPU resources with a higher number of VMs
  • Unstable performance

An alternative approach to overcome the above flaws of virtualization is to Containerize the applications and the run-time environment together.

What is containerization  

Containerization is an OS-level virtualization; wherein the entire build of an application along with run-time environment is encapsulated or bundled up in a package.

These packages are called containers.

Containers are lightweight virtualized environments. These are independent of the infrastructure both hardware and software.

The run-time environment includes the operating system, binaries, libraries, configuration files and other applications as shown in Diagram (9).

Packaged code

What is Dockers

Dockers provide an excellent framework for containerization and allow to build, ship, and run distributed applications over multiple platforms.

Docker framework is setup as a docker engine installed on host OS and a docker daemon (background process) process is started that manage the virtual containers.

Docker architecture

Refer Diagram (10) that shows a Docker engine with 3 containers residing on host OS (MAC OS).

An instruction file called dockerfile is written with a set of system commands that change the filesystem such as add, copy or delete commands, run commands, install utilities, system calls etc…

This dockerfile is built and packaged along with its run-time environment as an executable file called a docker image.

Docker daemon services run these images to create docker containers.

Docker container is a run-time instance of an image

It is wise to say that many images (or layers of instruction files) make up a container.

Docker containers have a compact packaging and each container is well isolated.

We can run, start, stop, attach, move or delete containers as these runs as services on the host OS.

Each image is made up of different layers; each image based on top of the other with the customized command changes that we make.

Every time we make a change in the filesystem, each change related to the image is encapsulated in a new layer of filesystem and stacked up above the parent image.

Only the changed layers are rebuilt, rest of the unchanged image layers are reused.

Certain docker commands ADD, RUN and COPY create a new layer with increased byte size; rest of the commands simply adds up a new layer with zero-byte size.

These layers are re-used to build a new image, hence faster and lightweight.

Docker images are also

The layer approach of an image every time there is a change in the image makes it possible to Version control the docker images.

Here is a terminal recording that shows docker engine process and how images and containers are created.

Docker documentation - to create containers.

Ppt diagram:

Code -> package -> build images -> registry hub -> download/pull image -> run container

Docker architecture

Animation: sheet4

Let’s consider the docker container: divyabhushan/learn_docker hosted on docker hub.

Latest tagged image: centOS_release1.2

What is the container environment?

Base OS: Centos:7

Utilities: vim, yum, git

Apps/files: Dockerfile, myApp.sh, runtests.sh, data and other supporting files.

Git source code: dockerImages

Download as: git clone https://github.com/divyabhushan/DockerImages_Ubuntu.git

What does the container do?
Container launches “myApp.sh” in Ubuntu:14.04 environment and run some scripts along with a set of post test_suites in the container (Ubuntu:14.04) and saves the output log file.

How to modify and build your own app

Step 1: pull 

1.1: Pull the docker image



1.2: Run image to create a container and exit


Step 2: modify

2.1: Start the container

2.2: Attach to the container and make some changes

Step 3: commit

3.1: Examine the history logs and changes in the container

3.2: Commit the changes in container

Step 4: push

4.1: Push new image to docker hub

Let us see the steps in action:

Step 1: pull 

docker image on your machine

1.1: Pull the docker image

Command:

docker pull divyabhushan/learn_docker:myApp_ubuntu_14.04

View the image on system

docker images

screenshot

Run image to create a container and exit

Command:

docker run -it --name ubuntu14.04 0a6f949131a6

Run command in ubuntu container and exit, the container is stopped on exiting out.

Docker Vs Virtual Machines(VMs)

View the stopped container with the ‘ps -a’ command.

Docker Vs Virtual Machines(VMs)

Step 2: modify

Start the container

Command:

docker start <container_id>

Docker Vs Virtual Machines(VMs)

Now the container is listed as a running process Attach to the container and make some changes

Command:

docker attach 7d0d0225778c

edit the ‘git configuration’ file and ‘myApp.sh’ script

Docker Vs Virtual Machines(VMs)

Container is modified and stopped

Step 3: commit

Examine the history logs and changes in the container

Docker Vs Virtual Machines(VMs)

The changes done inside the container filesystem can be viewed using the ‘docker diff’ command as:

Command: 

docker diff 7d0d0225778c

Docker Vs Virtual Machines(VMs)

Commit the changes in container

Docker commit:

Usage: docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]

docker commit -m 'new Ubuntu image' 7d0d0225778c divyabhushan/learn_docker:ubuntu14.04_v2

Docker Vs Virtual Machines(VMs)

New image is created and listed

Docker Vs Virtual Machines

Step 4: push

Push new image to docker hub

Command:

docker push divyabhushan/learn_docker:ubuntu14.04_v2

Docker Vs Virtual Machines

Point to note: just the latest commit change layer ‘50a5ce553bba’ has been pushed, while the other layers were re-used.

Image available on docker hub:

Docker Vs Virtual Machines

The latest tagged image can now be pulled from other machines; and run to create the same container environment.

Conclusion: An image was pulled and run to create a container to replicate the environment. Container was modified, new changes were committed to form a new image. New Image pushed back on the docker hub and now available as a new tag ready to be pulled by other machines.

Difference between Dockers and Virtual machines

Tabular differences on various parameters

ParametersVMsDockers
architecture

Hardware level virtualization. Each VM has its own copy of OS.

Software level virtualization. Dockers have no own OS, run on host OS


IsolationFully isolatedProcess or application-level isolation.  
Installation

Hypervisor can run directly on the hardware resources or on the host OS.


Docker engine is installed on top of the host OS. A docker daemon process is initiated on the host OS. There is no separate OS for every container.


CPU processing + performance


Slower: A VM contains the entire run-time environment that has to be loaded every time. Uses more CPU cycles; gives unstable performance.


Faster: Docker images are pre-built and share host resources as a result running an image as a container is lightweight and consumes less CPU cycle; gives a stable performance


Hardware storage


More storage space as each VM is an independent machine (OS). Example: 3 VMs of 800MB each will take 2.4 GB of space.Docker containers are lightweight since do not require to load OS+drivers, run on host OS as processes.
PortableDependency on host OS and hardware makes VM less portable. Importing a VM still requires manual setup such storage, RAM and network.Highly portable since lightweight and zero dependency on hardware.
Scalable and code-reusabilitySpinning up more VMs still need administrative tasks such as distributing resources to VM. Running a new machine puts extra load on the system resources also re-managing earlier VMs becomes a task. Every VM keeps its own copy of resources-poor code-reusability.Spinning up new docker containers simply means running pre-built images into containers as a process inside host OS. Containers are also configured on-the-fly passing parameters and run-time. Single image can be run and used to create many containers; encourage code-reusability
Resource utilizationStatic allocation results in resource wastage in case of idle VMs or if a VM’s resource requirement increases.Resources are dynamically allocated and de-allocated on the need basis by the docker engine.
Docker system prune or garbage collection

Virtual machines do not have an in-built prune mechanism, these have to be administered manually.


Docker image and containers can be pruned; which frees up a sensible amount of storage and memory space and CPU cycles.
New environmentCreating new VM from the scratch is a tedious, repetitive tasks. It involves installing a new OS, loading kernel drivers and other tools and configurations.Package the code and dependency files, build into an image, run the image to create a new container. Use an existing or a base image (dockerhub- scratch) to run and create more containers on the go.
Web-hosted HubNo web hosted hub for VMsdockerHub provides an open-source reliable trusted source of pre-built images that can be downloaded to run new containers.

Version control (backup, restore,track history)

(refer git)
Snapshot of VMs are not very user-friendly and consume more space.

Docker images are version controlled. 

Every delta difference in each docker container can easily be viewed (demo: docker diff <container_id>). 

Any change in the image is stored as a different layered version. A reference link to older images saves build time and space.

Auto-buildAutomation of creating VMs is not very feasible.Docker images can also be auto-built from every source code check-in to GitHub (Automated builds on Dockerhub)
Disaster recoveryTedious to recover from VM backup files.Easier to restore docker images (like files) just like git source files in case images are version controlled. Backup images only have to be run to create containers. (refer: screenshot).
UpdateAll the VMs have to updated with the release patch.A single image is updated, re-built and distributed across multiple platforms.
Memory usage+speedSlower: Entire snapshot of a machine and the OS is loaded into the cache memory.Real-time and fast: pre-built images. Only the instance, i.e, a container has to be run as a process and uses memory like an executable
Data integrityVM behavior may change if the dependency includes beyond the VM boundaries. (example: an app depends on production host network settings)Same behavior of apps in any environment
securityMore secure: A failure inside a VM may reach its guest OS but not the host OS or other virtual machines. Type-2 hypervisor though has a risk of kernel attack.Less secure: If a docker container compromised; underlying OS and hence all the containers may be affected since they share the same host kernel. OS Kernel may also be risked.
Key providersRed hat KVM, VMWare, Oracle VM VirtualBox, Mircrosoft Hyper-V, Citrix XenServerDockersGoogle kubernetes EngineAWS Elastic Container service
Data authenticationLot of software licenses.Docker maintains inbuilt content trust to verify published images.

Architecture comparison

 When to use VM or a Docker

When the need is an isolated OS, go for VMs.

For a hardware and software independent isolated application that needs fast distribution on multiple environments, use dockers.

  • Docker use-case:

Example: A database application along with its database

Consider the docker image - Oracle WebLogic Server on Docker Hub.

This image is pre-built Oracle WebLogic Server runtime environment, including Oracle Linux 7 and Oracle JDK 8 for deploying Java EE applications.

To create Server configurations on any machine, just download this image and run to create and start a container.

There is no need to install and configure JDK, Linux or other run-time environment.

  • Do not use Docker use-case:

The application depends on utility outside the docker container.

Code developed on dev machine with base OS as MAC; needs certain firewall setting on say Ubuntu OS.

How can the code be tested on the production ubuntu OS firewall while running from MAC OS docker container?

Solution:  Install a virtualization software on host OS-MAC; Create a VM (Virtual machine) with host OS as Ubuntu (same as production environment).

Configure the desired firewall settings on host VM – Ubuntu; import the test code inside Ubuntu and test.

  • Use a VM:

For Embedded systems programming, a VM is installed that connects to the system device drivers, controllers and kernel.

  • Virtualization used along with docker:

An extension to the previous scenario would be if you would want to also test your python application in the host OS-Ubuntu VM without having to set up the python exe and its libraries and binaries.

All you have to do is: Install Docker engine for Ubuntu OS and pull the python image from Docker hub as:

docker pull python:tag [ tag is the python version-choose the appropriate version ]

docker pull python:2.7

Refer: Python image

Either write a Dockerfile to import/copy entire source code to python environment or directly run the image passing the script path as below:

Command:

$docker run -it --name my-python-script -v “$PWD”:/usr/src/myapp -w /usr/src/myapp python:2.7 python my-application.py

Command options:

-v = volume list-bind mount a volume [mount present working directory onto /usr/src/myapp inside container]

-w = workdir string-working directory inside the container

Moreover; you can also test your python code in more than one version by downloading different python images, running them to create different containers and running your app in each container.

What’s exciting here is that once the code tested in each python environment; you could quickly work on the test results and drop the containers. And deploy the code to production only once code tested against various python versions.

Final thoughts

VMs and dockers are compatible with each other. Dockers are not here to replace Virtual machines.

Both serve the same purpose of virtualizing the computing and infrastructure resources for optimized utilization.

Using both Virtual machines and dockers together can yield better results in virtualization.

When one desires a fast, lightweight, portable and highly scalable hardware-independent environment for multiple applications isolation; wherein security is not the major concern; Dockers is the best choice.

Use a VM for embedded systems that are integrated with hardware; such as device driver or kernel coding.

A scenario simulating an infrastructure setup with a high resource control and dependency on system resources; VMs are a better choice.

Use of Dockers inside VM

CI/CD pipelines scenario:

Virtualization enables a smooth CI/CD process flow by promoting the users to concentrate only on developing the code on a working system that is set up for automated continuous integration and deployment without having to duplicate the entire setup each time.

A virtualized environment is set up; either using a VM or a docker image that takes care of the automatic code check-ins, builds, regression testing, and deployments on the server.

Divya

Divya Bhushan

Content developer/Corporate Trainer

  • Content Developer and Corporate Trainer with a 10-year background in Database administration, Linux/Unix scripting, SQL/PL-SQL coding, Git VCS. New skills acquired-DevOps and Dockers.
  • A skilled and dedicated trainer with comprehensive abilities in the areas of assessment, 
requirement understanding, design, development, and deployment of courseware via blended environments for the workplace. 

  • Excellent communication, demonstration, and interpersonal skills.

Website : https://www.knowledgehut.com/tutorials/git-tutorial

Join the Discussion

Your email address will not be published. Required fields are marked *

3 comments

saurabh 13 May 2019 1 likes

Well Written, thanks

Navneet 14 May 2019 1 likes

Excellent article ... concept of dockers is well articulated and explained.

Hugo 21 Jun 2019

I absolutely love your blog and find the majority of your post's to be exactly what I'm looking for

Suggested Blogs

How to Install Docker on Ubuntu

Docker is a platform that packages the application and all its dependencies in the container so that the application works seamlessly. The Container makes the application run its resource in an isolated process similar to the virtual machines, but it is more portable. For a detailed introduction to the different components of a Docker container, you can check out Introduction to Docker, Docker Containers & Docker Hub This tutorial covers the installation and use of Docker Community Edition (CE) on an Ubuntu 20.04 machine. Pre-requisitesAudienceThis tutorial is meant for those who are interested in learning Docker as a container service System Requirements Ubuntu 20.04 64-bit operating system. (If Linux OS is not in system, we can run docker using Virtual Box, PFB the steps) A user account with sudo privileges An account on Docker Hub to pull or push an image from Hub. Ubuntu Installation on Oracle Virtual Box If you want to use Ubuntu 20.04 without making any change to the Windows Operating system, you can proceed with the Oracle Virtual box.  Virtual Box is free and open-source virtualization software from Oracle. It enables you to install other operating systems in virtual machines. It is recommended that the system should have at least 4GB of RAM to get decent performances from the virtual operating system. Below are the steps for downloading Ubuntu 20.04 on Oracle Virtual box:Navigate to the website of Oracle Virtual Box, download the .exe file and get the latest stable version. 1. Once done with downloading the virtual box, we can navigate to and download the  Ubuntu disk image (.iso file) by clicking on the download option 2. Once the download has been completed for Ubuntu .iso file, open the virtual box and click on "New" present on top.  3. Enter the details of your virtual machine by giving any name, type as "Linux " and Version as Ubuntu (64 bit)  4. Choose the memory (RAM ) that needs to be allocated to the Virtual machine  and click on Next. (I have chosen 3000 MB) 5. After the RAM allocation ,Click on  Create a virtual disk now. This serves as the hard disk of the virtual Linux system. It is where the virtual system will store its files 6. Now, we want to select the Virtual Hard Disk.  7. We can choose either the “Dynamically allocated” or the “Fixed size” option for creating the virtual hard disk. 8. Finally, we have  to specify our Ubuntu OS's size. The recommended size is 10 GB, but it  can be increased if required.8. Finally, we have  to specify our Ubuntu OS's size. The recommended size is 10 GB, but it  can be increased if required.9. Ubuntu OS is ready to install in Virtual Box, but before starting the Virtual system, we need to a make few changes in settings. Click on storage under the setting.  10. Click on Empty under Controller IDE. Navigate to Attributes and browse the Optical Drive option. 11. Choose the .iso file from the location where it is downloaded. Once selected, click on OK and start the Virtual box by clicking on start present on the Top menu.12. Click ok and start the machine. 13. Proceed with "Install Ubuntu" 14. Under "Updates and other software" section, check "Normal installation", and the two options under “Other options” and continue.15. In Installation type, check Erase disk and install Ubuntu.16. Choose your current location and set up your profile. Click Continue.  17. It may take 10-15 minutes to complete the installation 18. Once the installation finishes, restart the virtual systemWe are done with pre-request, and can now proceed with using this Ubuntu. Docker Installation Process on Ubuntu  Method 1: Install Docker on Ubuntu Using Default Repositories One of the easiest ways is the installation of Docker from the standard Ubuntu 20.04 repositories, but It’s possible that the Ubuntu default repositories have not updated to the latest revision of Docker. It happens because in some cases Docker is not supporting that particular Ubuntu version. Therefore, there can be a scenario where  Ubuntu default repositories have not updated to the latest version. Log in to Virtual Box. Run “docker” as command to check if it is previously installed.To install Docker on Ubuntu box, first update the packages. It will ask for a password. Enter it and allow the system to complete the updates.sudo apt updateTo install Docker from Ubuntu default repositories, use the below command: sudo apt install docker.io To check the installed version, use the below: docker --version Since discussed above, it has installed the 19.03.8 version of docker whereas the latest version is 20.04  Method 2: Install Docker from Official Repository For installing docker on ubuntu 20.04 with the latest version, we’ll proceed with enabling the Docker repository, importing the repository GPG key, and finally installing the package. To install the docker on Ubuntu box, update your existing list of packages. It will ask for a password. Enter it and allow the system to complete the updates. sudo apt update  We need to install a few prerequisite packages to add HTTPS repository : sudo apt install apt-transport-https ca-certificates curl software-properties-common Import the repository’s GPG key using the following curl command: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - Add the Docker APT repository to the system sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"Again, update the package database with the Docker packages sudo apt update Finally, install Docker using below command: sudo apt install docker-ce To check the installed version use below: docker --versionTo start, enable and check the status of docker, use below command: sudo systemctl  status docker  sudo systemctl  start  docker  sudo systemctl  enable  docker To check system wide information regarding docker installation, we use the command “docker info”. Information that is shown includes the kernel version, number of containers and unique images. The output will contain details as given below, depending upon the daemon running: Source:$ docker info  Client:   Context:    default   Debug Mode: true  Server:   Containers: 14    Running: 3    Paused: 1    Stopped: 10   Images: 52   Server Version: 1.13.0   Storage Driver: overlay2    Backing Filesystem: extfs    Supports d_type: true    Native Overlay Diff: false   Logging Driver: json-file   Cgroup Driver: cgroupfs   Plugins:    Volume: local    Network: bridge host macvlan null overlay   Swarm: active    NodeID: rdjq45w1op418waxlairloqbm    Is Manager: true    ClusterID: te8kdyw33n36fqiz74bfjeixd    Managers: 1    Nodes: 2    Orchestration:     Task History Retention Limit: 5    Raft:     Snapshot Interval: 10000     Number of Old Snapshots to Retain: 0     Heartbeat Tick: 1     Election Tick: 3    Dispatcher:     Heartbeat Period: 5 seconds    CA Configuration:     Expiry Duration: 3 months    Root Rotation In Progress: false    Node Address: 172.16.66.128 172.16.66.129    Manager Addresses:     172.16.66.128:2477   Runtimes: runc   Default Runtime: runc   Init Binary: docker-init   containerd version: 8517738ba4b82aff5662c97ca4627e7e4d03b531   runc version: ac031b5bf1cc92239461125f4c1ffb760522bbf2   init version: N/A (expected: v0.13.0)   Security Options:    apparmor    seccomp     Profile: default   Kernel Version: 4.4.0-31-generic   Operating System: Ubuntu 16.04.1 LTS   OSType: linux   Architecture: x86_64   CPUs: 2   Total Memory: 1.937 GiB   Name: ubuntu   ID: H52R:7ZR6:EIIA:76JG:ORIY:BVKF:GSFU:HNPG:B5MK:APSC:SZ3Q:N326   Docker Root Dir: /var/lib/docker   Debug Mode: true    File Descriptors: 30    Goroutines: 123    System Time: 2016-11-12T17:24:37.955404361-08:00    EventsListeners: 0   Http Proxy: http://test:test@proxy.example.com:8080   Https Proxy: https://test:test@proxy.example.com:8080   No Proxy: localhost,127.0.0.1,docker-registry.somecorporation.com   Registry: https://index.docker.io/v1/   WARNING: No swap limit support   Labels:    storage=ssd    staging=true   Experimental: false   Insecure Registries:    127.0.0.0/8   Registry Mirrors:     http://192.168.1.2/     http://registry-mirror.example.com:5000/   Live Restore Enabled: false Note: In case you get below error after running “docker info” command, one way is to add sudo in front and run the command, OR you can refer to the same error-resolving steps mentioned under Running Docker Images section. Running Docker Images and Verifying the process: To check whether you can access and download the images from Docker Hub, run the following command: sudo docker run hello-worldIn case of errors received after running the docker run command, you can correct it using the following steps, otherwise proceed with the next step of checking the image. ERROR: docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.35/containers/create: dial unix /var/run/docker.sock: connect: permission denied. See 'docker run --help'.   Create the docker group if it does not exist sudo groupadd docker Add your user to the docker group.   sudo usermod -aG docker $USER   Eg:- sudo usermod -aG docker kanav Run the following command or Logout and login again and run ( if that doesn't work you may need to reboot your machine first)  newgrp docker Check if docker can be run without root docker run hello-world If the problem still continues, try to reboot it and run the command. To check the image, use this command: sudo docker images Uninstall Procedure: Below are the common commands used to remove images and containers: sudo  apt-get  purge docker-ce docker-ce-cli containerd.io To completely uninstall Docker, use below: To identify what are the installed packages, this is the command: dpkg -l | grep -i dockersudo apt-get purge -y docker-engine docker docker.io docker-ce docker-ce-cli  sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce   To remove images, containers, volumes, or user created configuration files, these commands can be used: sudo rm -rf /var/lib/docker /etc/docker sudo rm /etc/apparmor.d/docker sudo groupdel docker sudo rm -rf /var/run/docker.sock  Conclusion: If you found this Install Docker on Ubuntu blog relevant and useful, do check out the Docker-Training workshop from KnowledgeHut, where you can get equipped with all the basic and advanced concepts of Docker! 
5454
How to Install Docker on Ubuntu

Docker is a platform that packages the application... Read More

How to Install Kubernetes on Windows

Kubernetes is a container-based platform for managing cloud resources and developing scalable apps. It is widely regarded as the most common platform for automating, deploying, and scaling the entire cloud infrastructure. The platform runs on all major operating systems and is the most widely used open-source cloud tool.  Kubernetes can scale your entire infrastructure, monitor each service's health, act as a load balancer, and automate deployments, among other things. You can deploy your pods (docker containers) and services across the cloud by installing and configuring as many nodes (clusters) as you want.Let’s get started. We will guide you through the complete roadmap on how to install Kubernetes for Windows users. This tutorial will show you how to set up Kubernetes and deploy the official web GUI dashboard, which will allow you to manage and monitor everything. PrerequisitesFor installing Kubernetes in your system, here are a few prerequisites that need special attention. The hardware and software requirements are discussed below:Hardware requirementsMaster node with at least 2 GB memory. (Additional will be great)Worker node with 700 MB memory capacity.Your Mouse/Keyboard (monitor navigation)Software requirementsHype-VDocker DesktopUnique MAC addressUnique product UUID for every nodeEnsuring that there is a full range of connectivity between all the machines in the cluster is a must.Installation ProcedureStep 1: Install & Setup Hyper-VAs we all know, Windows has its virtualization software, known as Hyper-V, which is essentially VirtualBox on steroids. Hyper-V allows you to manage your virtual machines (VMs) using either the free Microsoft GUI tool or the command line. It's simple to enable Hyper-V, but first, make sure your PC meets the following requirements:Your operating system should be Windows 10 (Enterprise, Pro, or Education), withAt least 4GB of RAM and CPU Virtualization support, though you should double-check that it's turned on in your BIOS settings.You can disable or enable features like Hyper-V that may not be pre-installed when Windows is installed. Always keep in mind that some of the features require internet access to download additional Windows Update components.To enable Hyper-V on your machine, follow the steps below:1. Open the Control Panel.2. Select Programs from the left panel.3. Next, go to Programs and Features, then Turn Windows Features On or Off.4. Examine Hyper-V and the Hypervisor Platform for Windows.5. Select OK.Your system will now begin installing Hyper-V in the background; it may be necessary to reboot a few times until everything is properly configured. Don't hold your breath for a notification or anything! Verify that Hyper-V is installed successfully on your machine by running the following command as Administrator in PowerShell:Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-VOnce the state is shown as Enabled for above command in Power shell, we are good to go.Step 2: Download Docker for Windows and install it.Kubernetes is a container orchestration system built on top of Docker. It is essentially just a tool for communicating with Docker containers and managing everything at an enterprise level. Simply go to install Docker and click to Get Docker Desktop for Windows (stable).Windows users can use Docker Desktop.Docker Desktop for Windows is a version of Docker optimized for Windows 10. It's a native Windows application that makes developing, shipping, and running dockerized apps simple. Docker Desktop for Windows is the fastest and most reliable way to develop Docker apps on Windows, as it uses Windows-native Hyper-V virtualization and networking. Docker Desktop for Windows can run Docker containers on both Linux and Windows.Installation of Docker DesktopLet us take a look on the different steps involved in installing docker desktop.Double-click Docker for Windows Installer to run the installer.Docker starts automatically once the installation is complete. Docker is running and accessible from a terminal, as indicated by the whale in the notification area.Run Try out some Docker commands in a command-line terminal like PowerShell!  Run the Docker version to check the version.Run Docker run hello-world to verify that Docker can pull and run images.Boom!As long as the Docker Desktop for Windows app is running, Docker is accessible from any terminal. The Docker whale in the taskbar has a setting button that can be accessed from the UI.For a detailed step by step installation guide with screenshot, visit the blog - How to Install Docker on Windows, Mac, & Linux: A Step-By-Step GuideWARNING: FOLLOW THE INSTRUCTIONS BELOW! If Docker was successfully installed but you can't find its tray icon, you'll need to restart your computer. Check the official troubleshooting guide here if the issue persists. Step 3: Install Kubernetes on Windows 10Docker includes a graphical user interface (GUI) tool that allows you to change some settings or install and enable Kubernetes.To install Kubernetes, simply follow the on-screen instructions on the screen:1. Right-click the Docker tray icon and select Properties.2. Select "Settings" from the drop-down menu.3. Select "Kubernetes" from the left panel.4. Check Enable Kubernetes and click "Apply"Docker will install additional packages and dependencies during the installation process. It may take between 5 and 10 minutes to install, depending on your Internet speed and PC performance. Wait until the message 'Installation complete!' appears on the screen. The Docker app can be used after Kubernetes has been installed to ensure that everything is working properly. Both icons at the bottom left will turn green if both services (Docker and Kubernetes) are running successfully and without errors.Example.Step 4: Install Kubernetes DashboardThe official web-based UI for managing Kubernetes resources is Kubernetes Dashboard. It isn't set up by default. Kubernetes applications can be easily deployed using the cli tool kubectl, which allows you to interact with your cloud and manage your Pods, Nodes, and Clusters. You can easily create or update Kubernetes resources by passing the apply argument followed by your YAML configuration file.Use the following commands to deploy and enable the Kubernetes Dashboard.1. Get the yaml configuration file from here.2. Use this to deploy it. kubectl apply -f .\recommended.yaml3. Run the following command to see if it's up and running.:kubectl.exe get -f .\recommended.yaml.txtStep 5: Access the dashboardThe dashboard can be accessed with tokens in two ways: the first is by using the default token created during Kubernetes installation, and the second (more secure) method is by creating users, giving them permissions, and then receiving the generated token. We'll go with the first option for the sake of simplicity.1. Run the following command PowerShell (not cmd)((kubectl -n kube-system describe secret default | Select-String "token:") -split " +")[1]2. Copy the generated token3. Runkubectl proxy.4. Open the following link on your browser: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/5. SelectToken & paste the generated token6. Sign InFinallyYou'll be able to see the dashboard and your cloud resources if everything is set up correctly. You can then do almost all of the "hard" work without having to deal with the CLI every time. You may occasionally get your hands dirty with the command line, but if you don't understand Docker and Kubernetes or don't have the time to manage your own cloud, it's better to stick with some PaaS providers that can be quite expensive.Kubernetes Uninstallation ProcessThe procedures for uninstalling cert-manager on Kubernetes are outlined below. Depending on which method you used to install cert-manager - static manifests or helm - you have two options.Warning: To uninstall cert-maneger, follow the same steps as you did to install it, but in reverse. Whether cert-manager was installed from static manifests or helm, deviating from the following process can result in issues and potentially broken states. To avoid this, make sure you follow the steps outlined below when uninstalling.Step 1: Before continuing, make sure that all user-created cert-manager resources have been deleted. You can check for any existing resources with the following command:$ kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespacesAfter you've deleted all of these resources, you can uninstall cert-manager by following the steps outlined in the installation guide.Step 2: Using regular manifests to uninstall.Uninstalling from a regular manifest installation is as simple as reversing the installation process and using the delete command.kubectl.2. Delete the installation manifests using a link to your currently running version vX.Y. Z like so:$ kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/vX.Y.Z/cert-manager.yamlStep 3: Uninstalling with Helm.1. Uninstalling cert-manager from a Helm installation is as simple as reversing the installation process and using the delete command on both the server and the client. kubectl and helm.$ helm --namespace cert-manager delete cert-manager2. Next, delete the cert-manager namespace:$ kubectl delete namespace cert-manager3. Finally, delete the cert-manger  CustomResourceDefinitions using the link to the version vX.Y.Z you installed:$ kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/vX.Y.Z/cert-manager.crds.yamlThe namespace is in the process of being terminated.The namespace may become stuck in a terminating state if it is marked for deletion without first deleting the cert-manager installation. This is usually because the APIService resource is still present, but the webhook is no longer active and thus no longer reachable.4. To fix this, make sure you ran the above commands correctly, and if you're still having problems, run:$ kubectl delete apiservice v1beta1.ConclusionIn this tutorial, we have explained in detail how to install Kubernetes with Hyper-V. Also, we have tackled what requirements we need, both in terms of the software and hardware. We have explained how to install Hyper-V and Docker on Windows 10.   It is important to note that the fundamental difference between Kubernetes and Docker is that Kubernetes is meant to run across a cluster and Docker is meant to run through nodes.   Kubernetes is also more extensive than Docker Swarm and is meant to coordinate a cluster of nodes at scale in production in an efficient manner. Each software is crucial to having a smooth installation process.   We finally looked at how to install and uninstall Kubernetes.
1700
How to Install Kubernetes on Windows

Kubernetes is a container-based platform for manag... Read More

How To Install Jenkins on Ubuntu

Jenkins is a Java-built open-source Continuous Integration (CI) and CD platform. Basically, Jenkins builds software projects, checks and deploys them. This is one of the most practical programming tools you can master, and today we will show you how Jenkins is installed on Ubuntu 18.04. Use this powerful tool to activate your VPS server!Jenkins is loved by teams of all sizes, for different language projects like Java, Ruby, Dot Net, PHP etc. Jenkins is a platform that is autonomous, and can be used on Windows, Linux or any other operating system.  Prerequisites Hardware Requirements: RAM- 4 GB (Recommended) Storage- more than 50 GB of Hard Disk Space (Recommended)        Software Requirements: Java: Java Development Kit (JDK) or Java Runtime Environment (JRE).  Web Browser: Any browser such as Google Chrome, Mozilla Firefox, Microsoft Edge. Operating System: An Ubuntu 18.04 server installed with a non-root sudo user and firewall. For help in the planning of production capability of a Jenkins installation see Choosing the right hardware for Masters. Why Use Jenkins? You need to consider continuous integration (CI) and continuous delivery (CD) to understand Jenkins: Continuous integration – the practice of continuous production combined with the main industry.  Continuous delivery – the code is constantly delivered to an area after the code is ready for delivery. It could be for production or staging. The commodity is supplied to a consumer base that can provide QA or inspection by customers. Developers update the code regularly in the shared repository (such as GitHub or TFS). Improvements made in the source code are made at the end of the day, making it difficult to identify the errors. So, Jenkins is used here. Once a developer changes the repository, Jenkins will automatically enable the build and immediately warn you in the event of an error (Continuous Integration CI). Installation Procedure: Step 1: Install Java Skip to the next section if you have Java already installed on your system. To check, please run the following command in the terminal: java --version Jenkins needs Java for running, but it doesn't include certain distributions by default, and Java versions of Jenkins are incompatible. Multiple Java implementations are available to you. OpenJDK is currently the most popular one, which we will use in this guide. Being an open-source Java application, Jenkins requires the installation of OpenJDK 8 on your system. The apt repositories can directly access OpenJDK 8. The installation of OpenJDK from standard repositories is recommended. Open and enter the following in the terminal window: $ sudo apt update  $ sudo apt install openjdk-8-jdk The download and installation will be requested. Press the "Y" button and press the Enter button to finish the process. Java 8 will be installed on your system. We are ready to download Jenkins package now as we have our requirements ready! Step 2: Install Jenkins The default Ubuntu packages for Jenkins are always behind the current version of the project itself. You may use the project-maintained packages to install Jenkins to take advantage of the newest patches and features. 1. add the framework repository key: $ wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add  The device returns OK when the key is inserted. 2. Next, link the repository of Debian packages to the sources.list of the server: $ sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list' 3. When both are in place, upgrade to apt to use the new repository: $ sudo apt update 4. Install Jenkins: $ sudo apt install jenkins Now we're going to start the Jenkins server, as Jenkins and its dependencies are in place. Step 3: Start Jenkins 1. You can start Jenkins using systemctl: $ sudo systemctl start jenkins 2. As systemctl does not display performance, you can use the status command to check that Jenkins has successfully launched: $ sudo systemctl status jenkinsIf all went well, the start of the performance should demonstrate that the service is active and ready to boot: Output: jenkins.service - LSB: Start Jenkins at boot time     Loaded: loaded (/etc/init.d/jenkins; generated)     Active: active (exited) since Sat 2021-04-17 00:34:17 IST; 26s ago       Docs: man:systemd-sysv-generator(8)    Process: 17609 ExecStart=/etc/init.d/jenkins start (code=exited, status=0/SUCC As Jenkins is running, so adjust the firewall rules to complete our further setup of Jenkins from the web browser. Step 4: Opening the Firewall 1. Jenkins works by default on port 8080, so let's open the port with ufw: $ sudo ufw allow 8080  2. Check ufw’s status: $ sudo ufw status You will see that traffic from anywhere is permitted to port 8080. Output: Status: active  To                         Action      From  --                         ------      ----  8000                       ALLOW       Anywhere                    CUPS                       ALLOW       Anywhere                    27017                      ALLOW       Anywhere                    27017                      ALLOW       192.168.1.10                8080                       ALLOW       Anywhere                    8000 (v6)                  ALLOW       Anywhere (v6)               CUPS (v6)                  ALLOW       Anywhere (v6)               27017 (v6)                 ALLOW       Anywhere (v6)               8080 (v6)                  ALLOW       Anywhere (v6) 3. If the firewall is inactive, the following commands will allow OpenSSH and turn it back on: $ sudo ufw allow OpenSSH  $ sudo ufw enable We can finish the initial configuration with Jenkins installed and our firewall configured. Note: If you decide to continue to use Jenkins, use a Nginx Reverse Proxy at Ubuntu 18.04 to configure Jenkins with SSL when your exploration has been completed to protect your passwords and any sensitive system or product information sent between the machine and the server in plain text. Step 5: Setting Up Jenkins 1. To set up installation, visit Jenkins on its default 8080 port with your server domain name or IP address: http://your_server_ip_or_domain:8080 You should see the Unlock Jenkins screen, which displays the initial password's location:2. You can use the cat command to display the password: $ sudo cat /var/lib/jenkins/secrets/initialAdminPassword 3. Copy the alphanumeric terminal 32-character password and paste into the Administrator Password field, then click Continue. Output: 0aaaf00d9afe48e5b7f2a494d1881326 The following screen shows the ability to install or select certain plugins: 4. We will click on the option to install proposed plugins to start the installation process immediately. 5. When the installation is done, the first administrative user will be prompted. You can save this step and use your initial password to continue as an Admin. However, we will take some time to create the user. The Jenkins default server is NOT encrypted to prevent data from being protected. Use the Nginx Reverse Proxy on Ubuntu 18.04 to configure Jenkins with SSL. This protects the information of users and builds transmitted through the web interface. 6. You will see a configuration instance page, which asks you to confirm your Jenkins instance's URL of choice. Confirm either your server's domain name or the IP address of your server.  7. Click Save and Finish once you have confirmed the relevant information. A confirmation page will show you that "Jenkins is ready!"  Hit Start using Jenkins button and it will take you to the Jenkins dashboard.  Congratulations! You have completed the installation of Jenkins. Step 6: Creation of New Build Jobs in Jenkins: The freestyle job is a highly versatile and user-friendly choice. It's easy to set up and many of its options appear in many other build jobs. For all projects, you can use it. Follow the following steps: You have to login to your Jenkins Dashboard by visiting2) Create New item: Click on the New Item on the left-hand side of the dashboard.3) Fill the project description: You can enter the job details as per your need.4) Source Code Management: Under source code management, enter the repository URL.You can also use a Local repository. 5) Build Environment: Now in the Build section, Click on the “Add build Setup” Select "Execute Windows batch command".Now, add the java commands. In this article, we have used javac HelloWorld.java and java HelloWorld.   6) Save the project: Click Apply and save the project. 7) Build Source Code and check its status: Click on “Build Now” on the left-hand side of the screen to create the source code. 8) Console Output: Select the build number and click on “Console Output” to check the status of the build run. When it shows success, it means that we have successfully run the HelloWorld program from the cGitHub Repository. In case of failure, you can check the job logs by clicking on failure icon and debug the root cause.Uninstall Jenkins Follow the instructions to uninstall Jenkins: $ sudo apt-get remove jenkins Uninstall Jenkins: $ sudo apt-get remove --auto-remove jenkins Purging your data: $ sudo apt-get purge jenkins or you can use: $ sudo apt-get purge --auto-remove jenkins Conclusion: Installing Jenkins on Ubuntu is really that easy. Jenkins has a low learning curve and so you can start to work with it as quickly as possible. In the above article we have learned how to install Jenkins in an Ubuntu machine where all the steps are explained clearly. In case you want to learn more about the core concepts of Jenkins Jobs, Pipelines, Distributed System, Plugins, and how to use Jenkins in depth you can enroll for our course Jenkins Certification Course. 
5417
How To Install Jenkins on Ubuntu

Jenkins is a Java-built open-source Continuous In... Read More

Useful links