DevOps Roadmap to Become a Successful DevOps Engineer
By Divya Bhushan
“DevOps is a combination of best practices, culture, mindset, and software tools to deliver a high quality and reliable product faster”Benefits of DevOps (Dev+Ops(SysAdmins plus Database Admins) DevOps agile thinking drives towards an iterated continuous development model with higher velocity, reduced variations and better global visualization of the product flow. These three “V”s are achieved with synchronizing the teams and implementing CI/CD pipelines that automate the SDLC repetitive and complex processes in terms of continuous integration of code, continuous testing, and continuous delivery of features to the production-like environment for a high-quality product with shorter release cycles and reduced cost.This ensures customer satisfaction and credibility.A streamlined process in place with the help of best practices and DevOps tools reduce the overhead, and downtime thus giving more opportunity for innovation. As a matter of fact, DevOps way of defining every phase (coding, testing, infrastructure provisioning, deployment, and monitoring) as code also makes it easier to rollback a versioned code in case of disaster recovery and make the environment easily scalable, portable and secure.“DevOps tools help you accomplish what you can already do but do not have time to do it.”1. What are the tasks of a DevOps Engineer?A Summary of day-to-day tasks carried out by a DevOps engineer -Design, build, test and deploy scalable, distributed systems from development through productionManage the code repository(such as Git, SVN, BitBucket, etc.) including code merging and integrating, branching and maintenance and remote repository managementManage, configure and maintain infrastructure systemDesign the database architecture and database objects and synchronize the various environmentsDesign implement and support DevOps Continuous Integration and Continuous Delivery pipelinesResearch and implement new technologies and practicesDocument processes, systems, and workflowsCreation and enhancement of dynamic monitoring and alerting solutions using industry-leading servicesContinuously analyse tasks that are performed manually and can be replaced by codeCreation and enhancement of Continuous Deployment automation built on Docker and Kubernetes.2. Who can become a DevOps Engineer?DevOps is a vast environment that fits almost all technologies and processes into it. For instance, you could come from a coding or testing background or could be a system administrator, a database administrator, or Operations team there is a role for everyone to play in a DevOps approach.You are ready to become a DevOps Engineer if you have the below knowledge and/expertise-You have a Bachelor’s or Master’s or BSC degree (preferably in Computer Science, IT, Engineering, Mathematics, or similar)Minimum 2 years of IT experience as a Software Developer with a good understanding of SDLC lifecycle with lean agile methodology (SCRUM)Strong background in Linux/Unix & Windows AdministrationSystem development in an Object-oriented or functional programming language such as Python / Ruby / Java / Perl / Shell scripting / Groovy or GoSystem-level understanding of Linux (RedHat, CentOS, Ubuntu, SUSE Linux), Unix (Solaris, Mac OS) and Windows ServersShell scripting and automation of routines, remote execution of scriptsDatabase management experience in Mongo/Oracle or MySQL databaseStrong SQL and PL/SQL scriptingExperience working with source code version control management like Git, GitLab, GitHub or SubversionExperience with cloud architectures, particularly Amazon Web Services(AWS) or Google cloud platform or Microsoft AzureGood understanding of containerization using Dockers and/or KubernetesExperience with CI/CD pipelines using Jenkins and GitLabKnowledge of data-centre management, systems management, and monitoring, networking & securityExperience in Automation/configuration management using Ansible, and/or Puppet and/or ChefKnow how to monitor your code using Configuration Monitoring tools such as Nagios or PrometheusBackground in Infrastructure and NetworkingExtensive knowledge about RESTful APIsA solid understanding of networking and core Internet protocols (e.g. TCP/IP, DNS, SMTP, HTTP, and distributed networks)Excellent written and verbal English communication skillsSelf-learner, team layer, willingness to learn new technologies and ability to resolve issues independently and deliver results.3. Roadmap to becoming a DevOps Engineer3.1 Learn a programming languageA programming language enables a user to interact and manage the system resources such as the kernel, device drivers, memory devices, I/O devices; also to write software.A well-written piece of code will be more versatile, portable, error-proof, scalable and optimized that will enhance your DevOps cycle letting you be more productive with a high-quality product. As a DevOps Engineer, you will have to use many software and plugins for a CI/CD pipeline, and you will be at your best if you have a good grip on some of the popular programming languages:1. Java : An object-oriented, general-purpose programming language. Goal – “Write once, run anywhere”, which is synonymous with the Dockers(or containerization) philosophy 2. C: Is a general-purpose procedural programming language, it supports structured programming3. C#: A general-purpose, multi-paradigm object-oriented programming (OOP) language4. Python: Python is an easy to learn, interpreted, high-level and powerful programming language with an object-oriented approach. Ideal for infrastructure programming and web development. It has a very clear syntax5. Ruby: Is an open-source dynamic OOP programming language with an elegant and easy syntax. This implements multiple multi-paradigm approaches.As you know, DevOps majorly emphasizes on automating the repetitive and error-prone tasks. You ought to know any of the popular scripting languages:6. Perl: Perl is a highly capable scripting programming language, with its syntax very similar to C7. Bash shell script: Powerful set of instructions in a single shell script file to automate repetitive and complex commands8. JavaScript: An interpreted scripting language to build websites9. PowerShell for windows: A cross-platform automation and configuration framework or tool, that deals with structured data, REST APIs and object models. It has a command-line tool.Good-to-know language:10. Go: Go is an open-source programming language developed by Google. It is used to build simple, reliable and efficient software3.2 Understand different OS conceptsAs a Software developer, you must be able to write code that can interact with the machine resources and have a sound understanding of the underlying OS you are dealing with.Knowing the OS concepts will help you be more productive in your programming.This gives you the ability to make your code faster, manage processes, interact with the input-output devices, communicate with the other OS, optimize the processing usage, memory usage and disk usage of your program.As a DevOps engineer with infrastructure role, setting up and managing servers, controllers and switches becomes easier if you understand the resources, processes, and virtualization concepts very well.To be able to administer the users and groups, file permissions and security you must know the filesystem architecture.Essential OS concepts a DevOps engineer must know include:I. Kernel managementKernel is the core element of any OS. It connects the system hardware with the software. It is responsible for memory, storage, and process managementII. Memory ManagementMemory management is the allocation/deallocation of system memory(RAM, cache, page) to various system resources and to optimize the performance of the systemIII. Device drivers managementA device driver is a software program that controls the hardware device of the machineIV. Resource managementThe dynamic allocation/deallocation of system resources such as kernel, CPU, memory, disk and so onV. I/O managementCommunication between various input/output devices connected to the machine such as- keyboard, mouse, disk, USB, monitor, printers, etc VI. Processes and process managementEvery program that executes a certain task is called a process, each process utilizes a certain amount of computational resources. The technique of managing various processes to share the load of memory, disk and CPU(processing) usage also the inter-process communication is termed as process managementVII. Threads and concurrencyMany programming languages support multi-threading and concurrency, i.e, the ability to run multiple tasks simultaneously VIII. Virtualization and containerizationConcept of simulating a single physical machine to multiple virtual machines/environments to optimize the use of resources and to reduce the time is taken and cost. Understand this well as you will often need to replicate the real-time environment.Linux containers are a great concept to isolate and package an application along with its run-time environment as a single entity.Run-time environment includes all its dependencies, binaries, configuration files and libraries. Dockers is a containerized command-line tool that makes it easier to create, run and deploy applications with containers.Using both Virtual machines and dockers together can yield better results in virtualizationIX. Distributed file systemsA client machine can access data located on a Server machine. This is true in the case of a client/server-based application model.X. Filesystem architectureThe architectural layout of how and in what hierarchy the data is organized on a disk, will make your task of managing data easier.3.3 Learn about managing serversAs cloud deployments become more useful with DevOps approach, there is a need to manage a group of Servers (Application, Database, Web Server, Storage, Infrastructure, Networking Server and so on) rather than individual servers.You should be dynamically scaled up/down the servers, without rewriting the configuration files.Nginx: This is a web server that can also be used as a reverse proxy, load balancer, mail proxy, and HTTP cache.This provides robust and customizable monitoring of your cloud instances and their status. Nginx offers more flexibility and configurability for better configuration and automation using DevOps tools like Puppet and Chef.3.4 Networking and SecurityIn a highly connected network of computers, it becomes essential to understand the basic concepts of networking, how to enforce security and diagnose problems.As a DevOps engineer, you would also be required to set up an environment to test networking functions. In addition, set up continuous integration, delivery and deployment pipelines for network functions.Learn the basic networking concepts like Ip addresses, DNS, routing, firewalls and ports, basic utilities like ping, ssh, netstat, ncr and ip, load balancing and TLS encryption.Understand the basic protocols(standard rules for networking) such as-TCP/IP (Transfer Control Protocol/Internet Protocol), HTTP (Hypertext Transfer Protocol), SSL, SSH (Secure Shell), FTP (File Transfer Protocol), DNS (Domain Name Server).Configuration management tools like Ansible and Jenkins can be used to configure and orchestrate network devices.3.5 What is and how to set-upAs a DevOps methodology we often describe CI/CD pipeline, let us understand what is it?Continuous Integration(CI) is a development practice wherein developers regularly merge or integrate their code changes into a commonly shared repository very frequently.If I speak from a VCS (preferably Git’s) point of view -Every minor code change done on various branches (from different contributors) is pushed and integrated with the main release branch several times a day, rather than waiting for the complete feature to be developed.Every code check-in is then verified by an automated build and automated test cases. This approach helps to detect and fix the bugs early, resolve the conflicts that may arise, improve software quality, reduce the validation and feedback loop time; hence increasing the overall product quality and speedy product releases.Continuous Delivery(CD) is a software practice where every code check-in is automatically built, tested and ready for a release(delivery) to production. Every code check-in should be release/deployment ready.CD phase delivers the code to a production-like-environment such as dev, uat, preprod, etc and runs automated tests.On successful implementation of continuous delivery in the prod-like environment, the code is ready to be deployed to the main production server.It is best to learn the DevOps lifecycle of continuous development, continuous build, continuous testing, continuous integration, continuous deployment and continuous monitoring throughout the complete product lifecycle.Based on the DevOps process setup use the right tools to facilitate the CI/CD pipeline.3.6 Learn Infrastructure as codeInfrastructure as code (IaC) is to define(or declare) and manage the infrastructure resources programmatically by writing code as configuration files instead of managing each resource individually.These infrastructure resources(hardware and software) may be set up on a physical server, a Virtual machine or cloud.An IaC defines the desired state of the machine and generates the same environment every time it is compiled.What does IaC do?Automation: Spinning up or scaling down many resources becomes easier, as just a configuration file needs to be compiled and run. This reduces the overhead and the time spent.Versioning: IaC is a text file which can be versioned controlled which means 3 things:Infrastructure changes such as scaling up/down the resources and or changing/updating the resources (filesystem or user management) can be tracked through the versioned historyConfiguration files are easily shareable and portable and are checked-in as source codeAn IaC text file can easily be scheduled to be run in a CI/CD pipeline for Server management and orchestration.Manual errors eliminated: productivity increasedEach environment is an exact replica of production.How to do it?Use tools like Puppet, Ansible, Chef, TerraformThese tools aim at providing a stable environment for both development and operations tasks that results in smooth orchestration.A. Puppet: Puppet is a Configuration Management Tool (CMT) to build, configure and manage infrastructure on physical or virtual machinesB. Ansible: is a Configuration management, Deployment and Orchestration toolC. Chef: is a configuration management tool written in Ruby and Erlang to deploy, manage, update and repair server and application to any environmentD. Terraform: This automation tool builds, change, version and improve infrastructure and servers safely and efficiently.How will IaC be applied in DevOps?IaC configuration files are used to build CI/CD pipelines.IaC definitions enable DevOps teams to test applications/software in production-like stable environments quickly and effortlessly.These environments with IaC are repeatable and prevent runtime issues caused due to misconfiguration or missing dependencies.---3.7 Learn some Continuous Integration and Delivery (CI/CD) toolsIn order to continuously develop, integrate, build, test, apply feedback, deliver our product features to the production environment or deploy to the customer site, we have to build an automated sequence of jobs(processes) to be executed using the appropriate tools.CI/CD pipeline requires custom code and working with multiple software packages simultaneously. As a DevOps Engineer, here are some widely used tools you must know-a. Jenkins is an open-source automation server. Using Jenkins plugins CI/CD pipelines are built to automatically build, test and deploy the source codeJenkins is a self-contained Java-based program and easy to configure, extensible and distributedb. GitLab CI is a single tool for the complete DevOps cycle. Every code check-ins trigger builds, run tests, and deploy code in a virtual machine or docker container or any other server. Its has an excellent GUI interface. GitLab CI also has features for monitoring and securityc. CircleCI software is used to build, test, deploy and automate the development cycle. This is a secure and scalable tool with huge multi-platform support for IOS and MAC OS using MAC virtual machines along with Android and Linux environmentsd. Microsoft VSTS(Visual Studio Team Services) is not only a CI/CD service but also provide unlimited cloud-hosted private code repositoriese. CodeShip tool empowers your DevOps CI/CD pipelines with easy, secure, fast and reliable builds with native docker support. It provides a GUI to easily configure the buildsf. Bamboo by Atlassian is a Continuous integration, deployment and delivery Server. Bamboo has built-in Jira Software and BitBucket Software Integration, also built-in git branching and workflows.Jenkins is the most popular and widely used tool with numerous flexible plugins that integrate with almost any CI/CD toolchain. Also the ability of Jenkins to automate any project really distinguish this tool from others, thus it is highly recommended to get a good grip of this tool as a DevOps practitioner.Note: Since this is also a key for enthusiasts to choose the right tool but should be short definitions3.8 Know the tools to monitor software and infrastructureIt is crucial to continuously monitor the software and infrastructure upon setting up the continuous integration and continuous delivery pipeline (CI/CD) to understand how well your DevOps setup is performing. Also, it is vital to monitor system events and get alerts in real-time. A hiccup in the pipeline such as an application dependency failure or a linking error, or say the database has a downtime must be immediately notable and taken care of.This is where a DevOps Engineer must be familiar with monitoring tools such as -1. Nagios: is an open-source software application that monitors systems, networks, and infrastructure(Servers) and generates logs and alerts2. Prometheus: is an open-source real-time metrics-based event monitoring and alerting system.3.9 Learn about Cloud ProvidersAs the computational need increases so do the demand of the infrastructure resources.Cloud computing is a higher level of virtualization, wherein the computing resources are outsourced on a “cloud” and available for use on a pay-as-you-go basis over the internet.Some of the leading cloud providers such as AWS, Google Cloud, Microsoft Azure to name a few provide varied cloud services like IaaS, PaaS, and SaaS.Begin part of a DevOps practice, you will often find the need to access various cloud services say for infrastructure resources, production-like environment on the go for testing your product without having to provision it, get multiple replicas of the production environment, create a failover cluster, backup and recover your database over the cloud and various other tasks.Some of the cloud providers and what they offer are listed below-A. AWS (Amazon Web Services): provide tooling and infrastructure resources readily available for DevOps programs customized as per your requirement. You can easily build and deliver products, automate CI/CD process without having to worry about provisioning and configuring the environmentB. Microsoft Azure: Create a reliable CI/CD pipeline, practice Infrastructure as Code and continuous monitoring through Microsoft-managed data centresC. Google Cloud Platform: Uses google-managed data centres to provide DevOps features like end-to-end CI/CD automation, Infrastructure as Code, configuration management, security management, and serverless computing.AWS is the most versatile and recommended provider that you may wish to start learning.4. What next after becoming a DevOps expert?“Sky is the only limit for a DevOps person !!!”Mastering the DevOps tools and practices opens up the door to new roles and challenges for you to learn and grow.4.1 DevOps EvangelistA technical Evangelist is a strong powerful and influential role that exhibits a strong thought process.A DevOps evangelist is a DevOps leader who identifies and implements the DevOps features to solve a business problem or a process, and then shares and promotes the benefits that come from DevOps practice.Also identifies the key roles and train the team in the same and is responsible for the success of entire DevOps processes and people.4.2 Code Release ManagerA Code Release Manager measures the overall progress of the project in terms of metrics, he/she is aware of the entire Agile methodology. A Release Manager is more involved in the coordination among all the phases of DevOps flow to support continuous delivery.4.3 Automation ArchitectThe key responsibility is to plan, analyze, and design a strategy to automate all manual tasks with the right tools and implement the processes for continuous deployment.4.4 Experience AssuranceAn experience Assurance person is responsible for the user experience and makes sure that the product being delivered meet the original business specifications.This role is also termed as Quality Assurance but with extended responsibilities of user experience testing. This role plays a critical role in the DevOps cycle.4.5 Software Developer/TesterUnder DevOps, the role and responsibilities of a Software Developer literally expand l, that the developers are no longer responsible for writing code, but also take ownership of unit testing, deployment and monitoring as well.A Developer/Tester has to make sure that the code meets the original business requirement.Henceforth; the role Developer/Tester or if the innovation extends further a Developer may also be referred to as DevTestOps.4.6 Security EngineerSecurity Engineer focuses on the Integrity of data by incorporating security into the product, and not at the end.He/she supports project teams in using security tools in the CI/CD pipeline, as well as provide resolution of identified security flaws. Conclusion“If you define the problem correctly, you almost have the solution.” - Steve JobsIn a nutshell, if you aspire to become a DevOps professional you ought to know -Programming language (C, Java, Perl, Python, Ruby, Bash shell, PowerShell)Operating System concepts (resource management)Source Control (like Git, Bitbucket, Svn, VSTS, etc)Continuous Integration and Continuous Delivery (Jenkins, GitLab CI, CircleCI)Infrastructure as Code (IaC) Automation (tools like Puppet, Chef, Ansible and/or Terraform)Managing Servers (application, storage, database, infrastructure, networking, web server etc)(Application, Database, Web Server, Storage, Infrastructure, Networking Server Networking and securityContainer Concepts (Docker)Continuous monitoring (Nagios and Prometheus)Cloud (like AWS, Azure, Google Cloud).DevOps ways( The three ways of DevOps) open the door of opportunities to improve and excel in the process using the right tools and technologies.“DevOps channels the entire process right from the idea on a whiteboard until the real product in the customer’s hands through automated pipelines(CI/CD).”As a DevOps Engineer you must be a motivated team player, need to have a desire to learn and grow, optimize the process and find better solutions.Since DevOps covers a vast area under its umbrella, it is best to focus on your key skills and learn the technologies and tools as needed.Understand the problem/challenge then find a DevOps solution around the same.
Rated 4.0/5
based on 44 customer reviews
Divya Bhushan
Content developer/Corporate Trainer
Content Developer and Corporate Trainer with a 10-year background in Database administration, Linux/Unix scripting, SQL/PL-SQL coding, Git VCS. New skills acquired-DevOps and Dockers.
A skilled and dedicated trainer with comprehensive abilities in the areas of assessment, requirement understanding, design, development, and deployment of courseware via blended environments for the workplace.
Excellent communication, demonstration, and interpersonal skills.
Posts by Divya Bhushan
DevOps Roadmap to Become a Successful DevOps Engineer
By Divya Bhushan
“DevOps is a combination of best practices, culture, mindset, and software tools to deliver a high quality and reliable product faster”Benefits of DevOps (Dev+Ops(SysAdmins plus Database Admins) DevOps agile thinking drives towards an iterated continuous development model with higher velocity, reduced variations and better global visualization of the product flow. These three “V”s are achieved with synchronizing the teams and implementing CI/CD pipelines that automate the SDLC repetitive and complex processes in terms of continuous integration of code, continuous testing, and continuous delivery of features to the production-like environment for a high-quality product with shorter release cycles and reduced cost.This ensures customer satisfaction and credibility.A streamlined process in place with the help of best practices and DevOps tools reduce the overhead, and downtime thus giving more opportunity for innovation. As a matter of fact, DevOps way of defining every phase (coding, testing, infrastructure provisioning, deployment, and monitoring) as code also makes it easier to rollback a versioned code in case of disaster recovery and make the environment easily scalable, portable and secure.“DevOps tools help you accomplish what you can already do but do not have time to do it.”1. What are the tasks of a DevOps Engineer?A Summary of day-to-day tasks carried out by a DevOps engineer -Design, build, test and deploy scalable, distributed systems from development through productionManage the code repository(such as Git, SVN, BitBucket, etc.) including code merging and integrating, branching and maintenance and remote repository managementManage, configure and maintain infrastructure systemDesign the database architecture and database objects and synchronize the various environmentsDesign implement and support DevOps Continuous Integration and Continuous Delivery pipelinesResearch and implement new technologies and practicesDocument processes, systems, and workflowsCreation and enhancement of dynamic monitoring and alerting solutions using industry-leading servicesContinuously analyse tasks that are performed manually and can be replaced by codeCreation and enhancement of Continuous Deployment automation built on Docker and Kubernetes.2. Who can become a DevOps Engineer?DevOps is a vast environment that fits almost all technologies and processes into it. For instance, you could come from a coding or testing background or could be a system administrator, a database administrator, or Operations team there is a role for everyone to play in a DevOps approach.You are ready to become a DevOps Engineer if you have the below knowledge and/expertise-You have a Bachelor’s or Master’s or BSC degree (preferably in Computer Science, IT, Engineering, Mathematics, or similar)Minimum 2 years of IT experience as a Software Developer with a good understanding of SDLC lifecycle with lean agile methodology (SCRUM)Strong background in Linux/Unix & Windows AdministrationSystem development in an Object-oriented or functional programming language such as Python / Ruby / Java / Perl / Shell scripting / Groovy or GoSystem-level understanding of Linux (RedHat, CentOS, Ubuntu, SUSE Linux), Unix (Solaris, Mac OS) and Windows ServersShell scripting and automation of routines, remote execution of scriptsDatabase management experience in Mongo/Oracle or MySQL databaseStrong SQL and PL/SQL scriptingExperience working with source code version control management like Git, GitLab, GitHub or SubversionExperience with cloud architectures, particularly Amazon Web Services(AWS) or Google cloud platform or Microsoft AzureGood understanding of containerization using Dockers and/or KubernetesExperience with CI/CD pipelines using Jenkins and GitLabKnowledge of data-centre management, systems management, and monitoring, networking & securityExperience in Automation/configuration management using Ansible, and/or Puppet and/or ChefKnow how to monitor your code using Configuration Monitoring tools such as Nagios or PrometheusBackground in Infrastructure and NetworkingExtensive knowledge about RESTful APIsA solid understanding of networking and core Internet protocols (e.g. TCP/IP, DNS, SMTP, HTTP, and distributed networks)Excellent written and verbal English communication skillsSelf-learner, team layer, willingness to learn new technologies and ability to resolve issues independently and deliver results.3. Roadmap to becoming a DevOps Engineer3.1 Learn a programming languageA programming language enables a user to interact and manage the system resources such as the kernel, device drivers, memory devices, I/O devices; also to write software.A well-written piece of code will be more versatile, portable, error-proof, scalable and optimized that will enhance your DevOps cycle letting you be more productive with a high-quality product. As a DevOps Engineer, you will have to use many software and plugins for a CI/CD pipeline, and you will be at your best if you have a good grip on some of the popular programming languages:1. Java : An object-oriented, general-purpose programming language. Goal – “Write once, run anywhere”, which is synonymous with the Dockers(or containerization) philosophy 2. C: Is a general-purpose procedural programming language, it supports structured programming3. C#: A general-purpose, multi-paradigm object-oriented programming (OOP) language4. Python: Python is an easy to learn, interpreted, high-level and powerful programming language with an object-oriented approach. Ideal for infrastructure programming and web development. It has a very clear syntax5. Ruby: Is an open-source dynamic OOP programming language with an elegant and easy syntax. This implements multiple multi-paradigm approaches.As you know, DevOps majorly emphasizes on automating the repetitive and error-prone tasks. You ought to know any of the popular scripting languages:6. Perl: Perl is a highly capable scripting programming language, with its syntax very similar to C7. Bash shell script: Powerful set of instructions in a single shell script file to automate repetitive and complex commands8. JavaScript: An interpreted scripting language to build websites9. PowerShell for windows: A cross-platform automation and configuration framework or tool, that deals with structured data, REST APIs and object models. It has a command-line tool.Good-to-know language:10. Go: Go is an open-source programming language developed by Google. It is used to build simple, reliable and efficient software3.2 Understand different OS conceptsAs a Software developer, you must be able to write code that can interact with the machine resources and have a sound understanding of the underlying OS you are dealing with.Knowing the OS concepts will help you be more productive in your programming.This gives you the ability to make your code faster, manage processes, interact with the input-output devices, communicate with the other OS, optimize the processing usage, memory usage and disk usage of your program.As a DevOps engineer with infrastructure role, setting up and managing servers, controllers and switches becomes easier if you understand the resources, processes, and virtualization concepts very well.To be able to administer the users and groups, file permissions and security you must know the filesystem architecture.Essential OS concepts a DevOps engineer must know include:I. Kernel managementKernel is the core element of any OS. It connects the system hardware with the software. It is responsible for memory, storage, and process managementII. Memory ManagementMemory management is the allocation/deallocation of system memory(RAM, cache, page) to various system resources and to optimize the performance of the systemIII. Device drivers managementA device driver is a software program that controls the hardware device of the machineIV. Resource managementThe dynamic allocation/deallocation of system resources such as kernel, CPU, memory, disk and so onV. I/O managementCommunication between various input/output devices connected to the machine such as- keyboard, mouse, disk, USB, monitor, printers, etc VI. Processes and process managementEvery program that executes a certain task is called a process, each process utilizes a certain amount of computational resources. The technique of managing various processes to share the load of memory, disk and CPU(processing) usage also the inter-process communication is termed as process managementVII. Threads and concurrencyMany programming languages support multi-threading and concurrency, i.e, the ability to run multiple tasks simultaneously VIII. Virtualization and containerizationConcept of simulating a single physical machine to multiple virtual machines/environments to optimize the use of resources and to reduce the time is taken and cost. Understand this well as you will often need to replicate the real-time environment.Linux containers are a great concept to isolate and package an application along with its run-time environment as a single entity.Run-time environment includes all its dependencies, binaries, configuration files and libraries. Dockers is a containerized command-line tool that makes it easier to create, run and deploy applications with containers.Using both Virtual machines and dockers together can yield better results in virtualizationIX. Distributed file systemsA client machine can access data located on a Server machine. This is true in the case of a client/server-based application model.X. Filesystem architectureThe architectural layout of how and in what hierarchy the data is organized on a disk, will make your task of managing data easier.3.3 Learn about managing serversAs cloud deployments become more useful with DevOps approach, there is a need to manage a group of Servers (Application, Database, Web Server, Storage, Infrastructure, Networking Server and so on) rather than individual servers.You should be dynamically scaled up/down the servers, without rewriting the configuration files.Nginx: This is a web server that can also be used as a reverse proxy, load balancer, mail proxy, and HTTP cache.This provides robust and customizable monitoring of your cloud instances and their status. Nginx offers more flexibility and configurability for better configuration and automation using DevOps tools like Puppet and Chef.3.4 Networking and SecurityIn a highly connected network of computers, it becomes essential to understand the basic concepts of networking, how to enforce security and diagnose problems.As a DevOps engineer, you would also be required to set up an environment to test networking functions. In addition, set up continuous integration, delivery and deployment pipelines for network functions.Learn the basic networking concepts like Ip addresses, DNS, routing, firewalls and ports, basic utilities like ping, ssh, netstat, ncr and ip, load balancing and TLS encryption.Understand the basic protocols(standard rules for networking) such as-TCP/IP (Transfer Control Protocol/Internet Protocol), HTTP (Hypertext Transfer Protocol), SSL, SSH (Secure Shell), FTP (File Transfer Protocol), DNS (Domain Name Server).Configuration management tools like Ansible and Jenkins can be used to configure and orchestrate network devices.3.5 What is and how to set-upAs a DevOps methodology we often describe CI/CD pipeline, let us understand what is it?Continuous Integration(CI) is a development practice wherein developers regularly merge or integrate their code changes into a commonly shared repository very frequently.If I speak from a VCS (preferably Git’s) point of view -Every minor code change done on various branches (from different contributors) is pushed and integrated with the main release branch several times a day, rather than waiting for the complete feature to be developed.Every code check-in is then verified by an automated build and automated test cases. This approach helps to detect and fix the bugs early, resolve the conflicts that may arise, improve software quality, reduce the validation and feedback loop time; hence increasing the overall product quality and speedy product releases.Continuous Delivery(CD) is a software practice where every code check-in is automatically built, tested and ready for a release(delivery) to production. Every code check-in should be release/deployment ready.CD phase delivers the code to a production-like-environment such as dev, uat, preprod, etc and runs automated tests.On successful implementation of continuous delivery in the prod-like environment, the code is ready to be deployed to the main production server.It is best to learn the DevOps lifecycle of continuous development, continuous build, continuous testing, continuous integration, continuous deployment and continuous monitoring throughout the complete product lifecycle.Based on the DevOps process setup use the right tools to facilitate the CI/CD pipeline.3.6 Learn Infrastructure as codeInfrastructure as code (IaC) is to define(or declare) and manage the infrastructure resources programmatically by writing code as configuration files instead of managing each resource individually.These infrastructure resources(hardware and software) may be set up on a physical server, a Virtual machine or cloud.An IaC defines the desired state of the machine and generates the same environment every time it is compiled.What does IaC do?Automation: Spinning up or scaling down many resources becomes easier, as just a configuration file needs to be compiled and run. This reduces the overhead and the time spent.Versioning: IaC is a text file which can be versioned controlled which means 3 things:Infrastructure changes such as scaling up/down the resources and or changing/updating the resources (filesystem or user management) can be tracked through the versioned historyConfiguration files are easily shareable and portable and are checked-in as source codeAn IaC text file can easily be scheduled to be run in a CI/CD pipeline for Server management and orchestration.Manual errors eliminated: productivity increasedEach environment is an exact replica of production.How to do it?Use tools like Puppet, Ansible, Chef, TerraformThese tools aim at providing a stable environment for both development and operations tasks that results in smooth orchestration.A. Puppet: Puppet is a Configuration Management Tool (CMT) to build, configure and manage infrastructure on physical or virtual machinesB. Ansible: is a Configuration management, Deployment and Orchestration toolC. Chef: is a configuration management tool written in Ruby and Erlang to deploy, manage, update and repair server and application to any environmentD. Terraform: This automation tool builds, change, version and improve infrastructure and servers safely and efficiently.How will IaC be applied in DevOps?IaC configuration files are used to build CI/CD pipelines.IaC definitions enable DevOps teams to test applications/software in production-like stable environments quickly and effortlessly.These environments with IaC are repeatable and prevent runtime issues caused due to misconfiguration or missing dependencies.---3.7 Learn some Continuous Integration and Delivery (CI/CD) toolsIn order to continuously develop, integrate, build, test, apply feedback, deliver our product features to the production environment or deploy to the customer site, we have to build an automated sequence of jobs(processes) to be executed using the appropriate tools.CI/CD pipeline requires custom code and working with multiple software packages simultaneously. As a DevOps Engineer, here are some widely used tools you must know-a. Jenkins is an open-source automation server. Using Jenkins plugins CI/CD pipelines are built to automatically build, test and deploy the source codeJenkins is a self-contained Java-based program and easy to configure, extensible and distributedb. GitLab CI is a single tool for the complete DevOps cycle. Every code check-ins trigger builds, run tests, and deploy code in a virtual machine or docker container or any other server. Its has an excellent GUI interface. GitLab CI also has features for monitoring and securityc. CircleCI software is used to build, test, deploy and automate the development cycle. This is a secure and scalable tool with huge multi-platform support for IOS and MAC OS using MAC virtual machines along with Android and Linux environmentsd. Microsoft VSTS(Visual Studio Team Services) is not only a CI/CD service but also provide unlimited cloud-hosted private code repositoriese. CodeShip tool empowers your DevOps CI/CD pipelines with easy, secure, fast and reliable builds with native docker support. It provides a GUI to easily configure the buildsf. Bamboo by Atlassian is a Continuous integration, deployment and delivery Server. Bamboo has built-in Jira Software and BitBucket Software Integration, also built-in git branching and workflows.Jenkins is the most popular and widely used tool with numerous flexible plugins that integrate with almost any CI/CD toolchain. Also the ability of Jenkins to automate any project really distinguish this tool from others, thus it is highly recommended to get a good grip of this tool as a DevOps practitioner.Note: Since this is also a key for enthusiasts to choose the right tool but should be short definitions3.8 Know the tools to monitor software and infrastructureIt is crucial to continuously monitor the software and infrastructure upon setting up the continuous integration and continuous delivery pipeline (CI/CD) to understand how well your DevOps setup is performing. Also, it is vital to monitor system events and get alerts in real-time. A hiccup in the pipeline such as an application dependency failure or a linking error, or say the database has a downtime must be immediately notable and taken care of.This is where a DevOps Engineer must be familiar with monitoring tools such as -1. Nagios: is an open-source software application that monitors systems, networks, and infrastructure(Servers) and generates logs and alerts2. Prometheus: is an open-source real-time metrics-based event monitoring and alerting system.3.9 Learn about Cloud ProvidersAs the computational need increases so do the demand of the infrastructure resources.Cloud computing is a higher level of virtualization, wherein the computing resources are outsourced on a “cloud” and available for use on a pay-as-you-go basis over the internet.Some of the leading cloud providers such as AWS, Google Cloud, Microsoft Azure to name a few provide varied cloud services like IaaS, PaaS, and SaaS.Begin part of a DevOps practice, you will often find the need to access various cloud services say for infrastructure resources, production-like environment on the go for testing your product without having to provision it, get multiple replicas of the production environment, create a failover cluster, backup and recover your database over the cloud and various other tasks.Some of the cloud providers and what they offer are listed below-A. AWS (Amazon Web Services): provide tooling and infrastructure resources readily available for DevOps programs customized as per your requirement. You can easily build and deliver products, automate CI/CD process without having to worry about provisioning and configuring the environmentB. Microsoft Azure: Create a reliable CI/CD pipeline, practice Infrastructure as Code and continuous monitoring through Microsoft-managed data centresC. Google Cloud Platform: Uses google-managed data centres to provide DevOps features like end-to-end CI/CD automation, Infrastructure as Code, configuration management, security management, and serverless computing.AWS is the most versatile and recommended provider that you may wish to start learning.4. What next after becoming a DevOps expert?“Sky is the only limit for a DevOps person !!!”Mastering the DevOps tools and practices opens up the door to new roles and challenges for you to learn and grow.4.1 DevOps EvangelistA technical Evangelist is a strong powerful and influential role that exhibits a strong thought process.A DevOps evangelist is a DevOps leader who identifies and implements the DevOps features to solve a business problem or a process, and then shares and promotes the benefits that come from DevOps practice.Also identifies the key roles and train the team in the same and is responsible for the success of entire DevOps processes and people.4.2 Code Release ManagerA Code Release Manager measures the overall progress of the project in terms of metrics, he/she is aware of the entire Agile methodology. A Release Manager is more involved in the coordination among all the phases of DevOps flow to support continuous delivery.4.3 Automation ArchitectThe key responsibility is to plan, analyze, and design a strategy to automate all manual tasks with the right tools and implement the processes for continuous deployment.4.4 Experience AssuranceAn experience Assurance person is responsible for the user experience and makes sure that the product being delivered meet the original business specifications.This role is also termed as Quality Assurance but with extended responsibilities of user experience testing. This role plays a critical role in the DevOps cycle.4.5 Software Developer/TesterUnder DevOps, the role and responsibilities of a Software Developer literally expand l, that the developers are no longer responsible for writing code, but also take ownership of unit testing, deployment and monitoring as well.A Developer/Tester has to make sure that the code meets the original business requirement.Henceforth; the role Developer/Tester or if the innovation extends further a Developer may also be referred to as DevTestOps.4.6 Security EngineerSecurity Engineer focuses on the Integrity of data by incorporating security into the product, and not at the end.He/she supports project teams in using security tools in the CI/CD pipeline, as well as provide resolution of identified security flaws. Conclusion“If you define the problem correctly, you almost have the solution.” - Steve JobsIn a nutshell, if you aspire to become a DevOps professional you ought to know -Programming language (C, Java, Perl, Python, Ruby, Bash shell, PowerShell)Operating System concepts (resource management)Source Control (like Git, Bitbucket, Svn, VSTS, etc)Continuous Integration and Continuous Delivery (Jenkins, GitLab CI, CircleCI)Infrastructure as Code (IaC) Automation (tools like Puppet, Chef, Ansible and/or Terraform)Managing Servers (application, storage, database, infrastructure, networking, web server etc)(Application, Database, Web Server, Storage, Infrastructure, Networking Server Networking and securityContainer Concepts (Docker)Continuous monitoring (Nagios and Prometheus)Cloud (like AWS, Azure, Google Cloud).DevOps ways( The three ways of DevOps) open the door of opportunities to improve and excel in the process using the right tools and technologies.“DevOps channels the entire process right from the idea on a whiteboard until the real product in the customer’s hands through automated pipelines(CI/CD).”As a DevOps Engineer you must be a motivated team player, need to have a desire to learn and grow, optimize the process and find better solutions.Since DevOps covers a vast area under its umbrella, it is best to focus on your key skills and learn the technologies and tools as needed.Understand the problem/challenge then find a DevOps solution around the same.
“DevOps is a combination of best practices, cul...
Read More
Docker Vs Virtual Machines(VMs)
By Divya Bhushan
Let’s have a quick warm up on the resource management before we dive into the discussion on virtualization and dockers.In today’s multi-technology environments, it becomes inevitable to work on different software and hardware platforms simultaneously.The need to run multiple different machines (Desktops, Laptops, handhelds, and Servers) platforms with customized hardware and software requirements has given the rise to a new world of virtualization in IT industry.What a machine need?Each computing environment(machine) needs its own component of hardware resources and software resources.As more and more machines are needed, building up and administering many such stand-alone machines is not only cumbersome, time consuming but also adds up to the cost and energy.Apparently; to run a customized High-power Scalable Server is a better idea to consolidate all the hardware and software requirements into one place and have a single server run and distribute the resources to many machines over a network.That saves us time, resources, energy and revenue.These gigantic servers are stored in a data warehouse called a Datacenter.Below Diagram (2) indicates a single server serving and sharing resources and data among multiple client machinesDoes this look simplified enough? Yes of course!So, this setup looks feasible we have a high-power, high-storage Server that gives resources to many smaller(resources) machines over a network.How to manage huge data - ServersWith Internet Of Things in boom, Information is overflowing with a huge amount of data; handling tremendous data needs more system resources which means more Dedicated servers are needed.Many Servers approach challenge:Running several Dedicated servers for specific services such as Web service, application or database service as indicated in Diagram (3) is difficult to administer and consumes more energy, resources, manpower and is highly expensive.In addition; resource utilization of servers is very poor resulting in resource wastage.This is where simulating different environments and running them all on a single server is a smart choice; rather than having to run multiple physically distinct servers.This is how Diagram (3) would change after consolidating different servers into one as shown in Diagram (4).Sheet 2VirtualizationWhat is VirtualizationThe above single server implementation can be defined as the following term.Virtualization is a technique used to simulate and pretend a single infrastructure resource (hardware resources and software resources) to be acting as many providing multiple functionalities or services without the need to physically build, install and configure.In other words;Running multiple simulated environments in a single machine without installing and configuring them is called Virtualization.Technically speaking;Virtualization is an abstract layer that shares the infrastructure resources among various simulated virtual machines without the need to physically set up these environments.Diagram (5) displays different virtual Operating systems are running on the same machine and using the same hardware architecture of the underlying machine.What is a Virtual machineThe simulated virtualized environments are called virtual machines or VM.Virtual machine is a replication/simulation of an actual physical machine.A VM acts like a real physical machine and uses the physical resources of the underlying host OS.A VM is a running instance of a real physical machine.Need for virtualizationSo; we have an overview of virtualization, let us examine when should we virtualize and what are the benefits of virtualization?Better resource management and cost-effective: as indicated in Diagram (6) and Diagram (7); hardware resources are distributed wisely on need basis to different environments; all the virtual machines share the same resources and reduce resource wastage.Ease of quick administration and maintenance: It is easier to build, install, configure one server rather than multiple servers. Updating a patch on various machines from a single virtualized server is much more feasible.Disaster recovery: Since all the virtualized machines reside on the same server and are treated as mounted volumes of data files, it is easier to back up these machines. In case of a disaster failure (power failure, network down, cyber-attacks, failed test code, etc) VM screenshots are used to recover the running state of the machine and the whole setup can be built up within minutes.Isolated and independent secure test environment: virtualization provide an isolated independent virtual test environment to test the legacy code or a vendor-specific product or even a beta release or say a corrupt code without affecting the main hardware and software platform. (This is a contradictory statement though; will discuss more under types of virtualization)These test environments like dev, uat, preprod, prod etc..can be easily tested and discarded.Easily scalable and upgradable: Building up more simulated environments means spinning up more virtual machines. Also upgrading VMs is as good as to run a patch in all VMs.Portable: Virtual machines are lightweight compared to the actual running physical machines; in addition, a VM that includes its own OS, drivers, and other installation files is portable on any machine. One can access the data virtually from any location.The screenshot of activity monitor below compares the CPU load:Implementation a) What is hypervisor and its types?As discussed in the previous section; virtualization is achieved by means of a virtualized layer on top of hardware or a software resource.This abstract layer is called a hypervisor.A hypervisor is a virtual machine monitor (VMM)There are 2 types of hypervisors: Diagram (8)Type-1 or bare-metal hypervisorType-2 or hosted hypervisorType-1 or bare-metal hypervisor is installed directly on the system hardware, thus abstracting and sharing the hardware components with the VMs.Type-2 or hosted hypervisor is installed on top of the system bootable OS called host OS; this hypervisor abstracts the system resources visible to the host OS and distributes it among the VMs.Both have their own role to play in virtualization.b) Comparing hypervisor typesType-1 or bare-metal hypervisorType-2 or hosted hypervisorInstalled directly on the infrastructure-OS independent and more secure against software issues.Installed on top of the host OS-more prone to software failures.Better resource flexibility: Have direct access to the hardware infrastructure (Hard-drive partition, RAM, embedded cards such as NIC). Provide more flexibility and scalability to the VMs and assign resources on a need basis.Limited resource allocation: Have access to just the resources exposed by the host OS.VMs installed will have limited access to hardware resources allocated and exposed by the host OS.Single point of failure: A compromised VM may affect the kernel. Extra security layers needed.A compromised VM may affect only the host OS, kernel still remains unreachable.Low latency due to direct link to the infrastructure.High latency as all the VMs have to pass through the OS layer to access the system resources.Generally used in ServersGenerally used on small client machinesExpensiveLess expensiveType-1 Hypervisors in market:VMWare ESX/ESXiHyperkit (OSX)Microsoft Hyper-V (Windows)KVM(Linux)Oracle VM ServerType-2 Hypervisors in market:Oracle VM VirtualBoxVMWare WorkstationParallels desktop for MACTypes of virtualizationBased on what resource is virtualized, there are different classifications of virtualization.Server, Storage device, operating system, networkDesktop virtualization: Entire desktop environment is simulated and distributed to run on a single server all at once. A desktop virtualization allows administrators to manage, install, configure similar setups on many machines. Upgrading all the machines with a single patch update or security checks becomes easier and faster.Server virtualization: Many dedicated servers can be virtualized into a single server that provides multi-server functionality.Example:Many virtual machines can be built up sharing the same underlying system resources.Storage, RAM, disks, CPUOperating system virtualization: This happens at the kernel level Hypervisor on hardware type 2 bare-metal One machine: Can boot up as multiple OS like Windows or Linux side-by-sideApplication virtualization: Apps are packaged and stored in a virtual environment and are distributed across different VMs. Example Microsoft applications like excel, MS word, Powerpoint etc, Citrix applications.Network functions virtualization: Physical network components such as NIC cards, switches, routers, servers, hubs, and cables are all assembled in a single server and used virtually by multiple machines without having the load of installing them on every machine.Virtualization is one of the building blocks and driving force behind cloud computing.Cloud computing provide virtualized need-based services. This has given an uplift to the concept of virtualization.A quick mention of various cloud computing models/services are listed below:SaaS – Software as a Service– end-user applications are maintained and run by service providers and easily distributed and used by the end users without having to install them.Top SaaS providers: Microsoft (Office suite, CRM, SQL server databases), AWS, Adobe, Oracle (ERP, CRM, SCM), Cisco’s Webex, GitHub ( git hosting web service)PaaS – Platform as a Service – computing infrastructure(hardware/software) is maintained and updated by the service provider and the user just have to run the product over this platform.Top Paas providers: AWS beanstalk, Oracle Cloud Platform (OCP), Google App EngineIaaS – Infrastructure as a Service – Provide infrastructure such as servers, physical storage, networking, memory devices etc. Users can build their own platform with customized operating system and applications.Key IaaS providers: Amazon Web Services, Microsoft Azure, Google compute engine, CitrixConclusion:We now have a fair understanding of types of virtualization and how they are implemented.ContainerizationThough virtualization has its pros; there are certain downsides of virtualization such as:Not all systems can be virtualized always.A corrupt VM is sometimes contagious and may affect other VMs or the kernel in-case of a Type-1 or bare-metal hypervisor.Latency of virtual disks due to increased payload on the CPU resources with a higher number of VMsUnstable performanceAn alternative approach to overcome the above flaws of virtualization is to Containerize the applications and the run-time environment together.What is containerization Containerization is an OS-level virtualization; wherein the entire build of an application along with run-time environment is encapsulated or bundled up in a package.These packages are called containers.Containers are lightweight virtualized environments. These are independent of the infrastructure both hardware and software.The run-time environment includes the operating system, binaries, libraries, configuration files and other applications as shown in Diagram (9).What is DockersDockers provide an excellent framework for containerization and allow to build, ship, and run distributed applications over multiple platforms.Docker framework is setup as a docker engine installed on host OS and a docker daemon (background process) process is started that manage the virtual containers.Refer Diagram (10) that shows a Docker engine with 3 containers residing on host OS (MAC OS).An instruction file called dockerfile is written with a set of system commands that change the filesystem such as add, copy or delete commands, run commands, install utilities, system calls etc…This dockerfile is built and packaged along with its run-time environment as an executable file called a docker image.Docker daemon services run these images to create docker containers.Docker container is a run-time instance of an imageIt is wise to say that many images (or layers of instruction files) make up a container.Docker containers have a compact packaging and each container is well isolated.We can run, start, stop, attach, move or delete containers as these runs as services on the host OS.Each image is made up of different layers; each image based on top of the other with the customized command changes that we make.Every time we make a change in the filesystem, each change related to the image is encapsulated in a new layer of filesystem and stacked up above the parent image.Only the changed layers are rebuilt, rest of the unchanged image layers are reused.Certain docker commands ADD, RUN and COPY create a new layer with increased byte size; rest of the commands simply adds up a new layer with zero-byte size.These layers are re-used to build a new image, hence faster and lightweight.Docker images are alsoThe layer approach of an image every time there is a change in the image makes it possible to Version control the docker images.Here is a terminal recording that shows docker engine process and how images and containers are created.Docker documentation - to create containers.Ppt diagram:Code -> package -> build images -> registry hub -> download/pull image -> run containerAnimation: sheet4Let’s consider the docker container: divyabhushan/learn_docker hosted on docker hub.Latest tagged image: centOS_release1.2What is the container environment?Base OS: Centos:7Utilities: vim, yum, gitApps/files: Dockerfile, myApp.sh, runtests.sh, data and other supporting files.Git source code: dockerImagesDownload as: git clone https://github.com/divyabhushan/DockerImages_Ubuntu.gitWhat does the container do?Container launches “myApp.sh” in Ubuntu:14.04 environment and run some scripts along with a set of post test_suites in the container (Ubuntu:14.04) and saves the output log file.How to modify and build your own appStep 1: pull 1.1: Pull the docker image1.2: Run image to create a container and exitStep 2: modify2.1: Start the container2.2: Attach to the container and make some changesStep 3: commit3.1: Examine the history logs and changes in the container3.2: Commit the changes in containerStep 4: push4.1: Push new image to docker hubLet us see the steps in action:Step 1: pull docker image on your machine1.1: Pull the docker imageCommand:docker pull divyabhushan/learn_docker:myApp_ubuntu_14.04View the image on systemdocker imagesscreenshotCommand:docker run -it --name ubuntu14.04 0a6f949131a6Run command in ubuntu container and exit, the container is stopped on exiting out.View the stopped container with the ‘ps -a’ command.Step 2: modifyStart the containerCommand:docker start Now the container is listed as a running process Attach to the container and make some changesCommand:docker attach 7d0d0225778cedit the ‘git configuration’ file and ‘myApp.sh’ scriptContainer is modified and stoppedStep 3: commitExamine the history logs and changes in the containerThe changes done inside the container filesystem can be viewed using the ‘docker diff’ command as:Command: docker diff 7d0d0225778cCommit the changes in containerDocker commit:Usage: docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]docker commit -m 'new Ubuntu image' 7d0d0225778c divyabhushan/learn_docker:ubuntu14.04_v2New image is created and listedStep 4: pushPush new image to docker hubCommand:docker push divyabhushan/learn_docker:ubuntu14.04_v2Point to note: just the latest commit change layer ‘50a5ce553bba’ has been pushed, while the other layers were re-used.Image available on docker hub:The latest tagged image can now be pulled from other machines; and run to create the same container environment.Conclusion: An image was pulled and run to create a container to replicate the environment. Container was modified, new changes were committed to form a new image. New Image pushed back on the docker hub and now available as a new tag ready to be pulled by other machines.Difference between Dockers and Virtual machinesTabular differences on various parametersParametersVMsDockersarchitectureHardware level virtualization. Each VM has its own copy of OS.Software level virtualization. Dockers have no own OS, run on host OSIsolationFully isolatedProcess or application-level isolation. InstallationHypervisor can run directly on the hardware resources or on the host OS.Docker engine is installed on top of the host OS. A docker daemon process is initiated on the host OS. There is no separate OS for every container.CPU processing + performanceSlower: A VM contains the entire run-time environment that has to be loaded every time. Uses more CPU cycles; gives unstable performance.Faster: Docker images are pre-built and share host resources as a result running an image as a container is lightweight and consumes less CPU cycle; gives a stable performanceHardware storageMore storage space as each VM is an independent machine (OS). Example: 3 VMs of 800MB each will take 2.4 GB of space.Docker containers are lightweight since do not require to load OS+drivers, run on host OS as processes.PortableDependency on host OS and hardware makes VM less portable. Importing a VM still requires manual setup such storage, RAM and network.Highly portable since lightweight and zero dependency on hardware.Scalable and code-reusabilitySpinning up more VMs still need administrative tasks such as distributing resources to VM. Running a new machine puts extra load on the system resources also re-managing earlier VMs becomes a task. Every VM keeps its own copy of resources-poor code-reusability.Spinning up new docker containers simply means running pre-built images into containers as a process inside host OS. Containers are also configured on-the-fly passing parameters and run-time. Single image can be run and used to create many containers; encourage code-reusabilityResource utilizationStatic allocation results in resource wastage in case of idle VMs or if a VM’s resource requirement increases.Resources are dynamically allocated and de-allocated on the need basis by the docker engine.Docker system prune or garbage collectionVirtual machines do not have an in-built prune mechanism, these have to be administered manually.Docker image and containers can be pruned; which frees up a sensible amount of storage and memory space and CPU cycles.New environmentCreating new VM from the scratch is a tedious, repetitive tasks. It involves installing a new OS, loading kernel drivers and other tools and configurations.Package the code and dependency files, build into an image, run the image to create a new container. Use an existing or a base image (dockerhub- scratch) to run and create more containers on the go.Web-hosted HubNo web hosted hub for VMsdockerHub provides an open-source reliable trusted source of pre-built images that can be downloaded to run new containers.Version control (backup, restore,track history)(refer git)Snapshot of VMs are not very user-friendly and consume more space.Docker images are version controlled. Every delta difference in each docker container can easily be viewed (demo: docker diff ). Any change in the image is stored as a different layered version. A reference link to older images saves build time and space.Auto-buildAutomation of creating VMs is not very feasible.Docker images can also be auto-built from every source code check-in to GitHub (Automated builds on Dockerhub)Disaster recoveryTedious to recover from VM backup files.Easier to restore docker images (like files) just like git source files in case images are version controlled. Backup images only have to be run to create containers. (refer: screenshot).UpdateAll the VMs have to updated with the release patch.A single image is updated, re-built and distributed across multiple platforms.Memory usage+speedSlower: Entire snapshot of a machine and the OS is loaded into the cache memory.Real-time and fast: pre-built images. Only the instance, i.e, a container has to be run as a process and uses memory like an executableData integrityVM behavior may change if the dependency includes beyond the VM boundaries. (example: an app depends on production host network settings)Same behavior of apps in any environmentsecurityMore secure: A failure inside a VM may reach its guest OS but not the host OS or other virtual machines. Type-2 hypervisor though has a risk of kernel attack.Less secure: If a docker container compromised; underlying OS and hence all the containers may be affected since they share the same host kernel. OS Kernel may also be risked.Key providersRed hat KVM, VMWare, Oracle VM VirtualBox, Mircrosoft Hyper-V, Citrix XenServerDockers, Google kubernetes Engine, AWS Elastic Container serviceData authenticationLot of software licenses.Docker maintains inbuilt content trust to verify published images. When to use VM or a DockerWhen the need is an isolated OS, go for VMs.For a hardware and software independent isolated application that needs fast distribution on multiple environments, use dockers.Docker use-case:Example: A database application along with its databaseConsider the docker image - Oracle WebLogic Server on Docker Hub.This image is pre-built Oracle WebLogic Server runtime environment, including Oracle Linux 7 and Oracle JDK 8 for deploying Java EE applications.To create Server configurations on any machine, just download this image and run to create and start a container.There is no need to install and configure JDK, Linux or other run-time environment.Do not use Docker use-case:The application depends on utility outside the docker container.Code developed on dev machine with base OS as MAC; needs certain firewall setting on say Ubuntu OS.How can the code be tested on the production ubuntu OS firewall while running from MAC OS docker container?Solution: Install a virtualization software on host OS-MAC; Create a VM (Virtual machine) with host OS as Ubuntu (same as production environment).Configure the desired firewall settings on host VM – Ubuntu; import the test code inside Ubuntu and test.Use a VM:For Embedded systems programming, a VM is installed that connects to the system device drivers, controllers and kernel.Virtualization used along with docker:An extension to the previous scenario would be if you would want to also test your python application in the host OS-Ubuntu VM without having to set up the python exe and its libraries and binaries.All you have to do is: Install Docker engine for Ubuntu OS and pull the python image from Docker hub as:docker pull python:tag [ tag is the python version-choose the appropriate version ]docker pull python:2.7Refer: Python imageEither write a Dockerfile to import/copy entire source code to python environment or directly run the image passing the script path as below:Command:$docker run -it --name my-python-script -v “$PWD”:/usr/src/myapp -w /usr/src/myapp python:2.7 python my-application.pyCommand options:-v = volume list-bind mount a volume [mount present working directory onto /usr/src/myapp inside container]-w = workdir string-working directory inside the containerMoreover; you can also test your python code in more than one version by downloading different python images, running them to create different containers and running your app in each container.What’s exciting here is that once the code tested in each python environment; you could quickly work on the test results and drop the containers. And deploy the code to production only once code tested against various python versions.Final thoughtsVMs and dockers are compatible with each other. Dockers are not here to replace Virtual machines.Both serve the same purpose of virtualizing the computing and infrastructure resources for optimized utilization.Using both Virtual machines and dockers together can yield better results in virtualization.When one desires a fast, lightweight, portable and highly scalable hardware-independent environment for multiple applications isolation; wherein security is not the major concern; Dockers is the best choice.Use a VM for embedded systems that are integrated with hardware; such as device driver or kernel coding.A scenario simulating an infrastructure setup with a high resource control and dependency on system resources; VMs are a better choice.Use of Dockers inside VMCI/CD pipelines scenario:Virtualization enables a smooth CI/CD process flow by promoting the users to concentrate only on developing the code on a working system that is set up for automated continuous integration and deployment without having to duplicate the entire setup each time.A virtualized environment is set up; either using a VM or a docker image that takes care of the automatic code check-ins, builds, regression testing, and deployments on the server.
