Chef

The “kitchen test” command is used to fully test the cookbooks in a CI/CD pipeline and is an elaborate process consisting of creating a new instance, converging it, verifying and then destroying the instance. Since this is an elaborate process it is not suited to be used in the development phase of a cookbook. "kitchen converge" and "kitchen verify" commands are more useful for the cookbook development phase.

“Kitchen converge” applies the cookbook to the existing sandbox environment. It will install the chef-client, load the cookbooks and other necessary files and goes ahead with the first Chef client run using kitchen.yml attributes. “Kitchen converge” has two types of return codes. A zero denotes success and a non zero value denotes failure at any step. If “kitchen converge” fails, try to delete the kitchen instance with “kitchen destroy” command and then run “kitchen converge” once again.

“Kitchen verify” helps in validating the state of the kitchen instance against test suites specified in .kitchen.yml file. Often "kitchen converge" and "kitchen verify" are frequently used to develop infrastructure code using Test Driven Development.

“metadata.rb” file is at the top level of every cookbook directory and gets created as soon as a cookbook is created. (For JAVA programmers the concept is similar to a constructor getting created when an object is created!) It specifies all the dependencies required to correctly deploy a cookbook to its nodes. The Chef server compiles this metadata.rb file into metadata.json file and stores it. We can edit the metadata in the metadata.rb file and should be recompiled every time by the Chef server for the changes to be implemented. Metadata.rb file is compiled when

1. A cookbook is uploaded to the Chef server
2. when the cookbook metadata knife subcommand is run and
3. when "knife cookbook create" subcommand is run, then the knife creates metadata.rb file automatically.

Metadata.rb files have entries under ‘depends’ keyword which specifies the different dependencies required to successfully deploy the cookbook in a specific node. If these dependencies are not given properly we may often face errors while deploying cookbooks and Chef nodes would not be configured as expected. Apart from "depends", we have other details given in the metadata.rb file like license, version, gem, etc. which can be parametrized using different operators (=,>=, >, etc)

From Chef 0.10, a ‘chefignore’ file is used to ignore unwanted temporary files like swap files, version control files, build data files, etc while uploading to the Chef server. This file is could be available at any of the subfolders of your cookbook repository. This file helps us to stop uploading unwanted files or confidential files to Chef Server.

For eg: while using Vim editor, it leaves behind swap files. These are of no use in reusing the uploaded cookbooks. These will get uploaded into the Chef Server unless specified in the chefignore file. The following entry in the chefignore file stops all swap files created by Vim editor being uploaded. This helps us to stop accidentally uploading these swap files which is not related to the cookbook and not required for the correct functioning of the cookbook.

# vim
*.sw[a-z]

Chefignore file can use *,** and? wildcards to specify the ignore patterns in chefignore file. The pattern matching in the file is concerning

• root of the cookbook
• Relative directory names

The different search indexes for node data in Chef Server are as follows:-

1. Data bags: Data bags are like a container where global data is stored and shared across the Chef nodes and these data are accessible by Chef Server. Data bags are used to securely store information like admin credentials etc in the form of JSON files. These are indexed and are loaded in a Chef search.
2. Environment: We at times want our nodes to converge to a testing environment or maybe to a production environment so that admins can know in which stage of the code development, the node is included. Environment details are also updated in Chef Server and could be searched for. “_default” is the default environment for any Chef node. Multiple environments could be added and worked upon.’
3. Node: Node is a system that we want to be configured and maintained using Chef Infra Client. The node could be a physical server or a virtual machine or a cloud machine. Details of the node are also updated in Chef Server and searched upon.
4. Role: Role is used for defining certain patterns or processes across all Chef nodes. Multiple configurations of the nodes at a different time could be achieved in Chef using the “roles” feature.

Recipes are the basic configuration element in any organisations. Its a programmatic representation of the desired state of the nodes. Usually written in ruby with Chef Domain Specific Language (DSL), these specify all the standards (policies) or updates that the nodes should follow. Recipes are often a collection of resources that are the building blocks of the infrastructure of any organisations.

Cookbooks are a collection of recipes. Apart from recipes, cookbooks contain other elements like metadata, attributes, libraries, resources, templates, and tests which helps the nodes to achieve their desired state. Cookbooks can be created by Chef commands or by the Chef command-line tool called Knife. The cookbooks are the fundamental units of chefs and are uploaded to the Chef Servers and then read and deployed by the Chef Clients at the nodes. Cookbooks are standalone entities that could be transferred easily between the Chef server and node. It typically implements a scenario of infrastructure and has everything required for the scenario. Cookbooks could be version controlled and hence help the development team to collaborate and share cookbooks.

The “knife upload” command is used to upload the cookbooks to the Chef server. It does not help in managing the cookbook dependencies. So knife command is not chosen for very complicated and large cookbooks with dependencies that in turn might be dependent on others and so on. For eg: Assume that a cookbook called “my-app” depends on community cookbook “nginx” which in turn depends on another community cookbook “ohai”. When we try to upload “my-app” to Chef Server using the “knife upload” command, we have to make sure all other dependent cookbooks are uploaded manually to Chef Server. This problem is solved using “berkshelf upload”

Berkshelf command is similar to knife command and helps to upload cookbooks to Chef Server. But it also helps in the management of cookbooks outside of Chef repository. This is quite useful in case of large and complex cookbooks with lots of dependencies. “berkshelf upload” will upload all dependent cookbooks to the Chef server automatically. All the dependent cookbooks are fetched using the “berks install” command of the Berkshelf tool.

InSpec is an in-built testing framework for testing and auditing infrastructure in Chef. A sample InSpec test to verify whether “Nginx” server is installed and running in a system could be written as follows:

```control 'Nginx-install-1.0' do
     title 'Check if nginx is installed'
     describe package 'nginx' do
      it { should be_installed }
     end
   end  
   control 'Nginx should be running' do
     describe service 'nginx' do
   it { should be_running }
     end
   end```

Main components of InSpec tests are:

• Control: A requirement is defined inside a control block
• Title: Title defines the metadata of the control block describing the importance of the requirement.
• Resource: Resources are the basic building blocks of infrastructure. A control block may have multiple resources. Resources could be anything like file, package, service, etc.
• Describe: Tests for a particular resource is grouped into a describe block. A control block should have a minimum one describe block. A describe block is defined within a “do end” block
• It or its: Each line starting with "it" or "its" is a test to access resource-specific matcher. “should be_installed” and “should be_running” are in-built methods in DSL to verify whether the package is installed and whether the service is running.

Chef Client run could be performed in the following modes:

1.  Local mode: In this mode, we can run the Chef Client against the local Chef repository mimicking the behaviour as if it was running against a Chef Server. This is achieved through ‘Chef zero’ (previously a standalone tool)which acts as a lightweight Chef Server. Running in local mode is the easiest way to quickly test and verify the infrastructure code, though this has some disadvantages of not testing the authentication or security checks which are needed when we connect to the actual Chef Server
2.  Why run mode: Why run mode gives you a preview of what will happen on your node when you perform Chef Client run. In why run mode no actual modifications are done on your node. This mode follows the principle of no-op(no operation). All actions that happen in normal Chef Client run is performed until the actual configuration of the node.
3.  Run in FIPS (Federal Information Processing Standards) mode: Chef Client run can be performed in FIPS mode to follow the FIPS standards. Any violation of these standards will throw a run exception.
4.  Run as a service/daemon: In this mode, we can make the Chef Client run happen in our desired time intervals. Chef Infra Client cookbook from GitHub repository is used to perform Chef Client run.
5.  Run with elevated privileges: When we are trying to configure nodes with different OS like Windows or Linux we often run into some errors while performing Chef Client run, this is mostly due to the access privileges on these systems. The client run solves this issue by providing the ability to run the Chef Client with root access rather than by a local system user
6.  Run as a non-root user: This mode is helpful when multiple teams (devs, administration, etc) are involved in your infrastructure development and not everyone needs the same amount of accesses. You can limit the access and make the Chef Client run as a non-root user.

We can update the Chef run lists with the below two options:

• The Chef run list can be updated using the knife commands:  Knife is a command-line utility tool that securely communicates to the Chef Server. We have different commands for adding, removing and setting recipes in the run list.

For eg: 

```knife node run_list add <NODE_NAME> <RUN_LIST_ITEM> (options)``` commands could be used to add a recipe or a role in the run list of a node.

Similarly 

```knife node run_list remove <NODE_NAME> <RUN_LIST_ITEM> (options)```  command could be used to remove from the run list.

We can also specify the run list for a node while bootstrapping the node using the “knife bootstrap” command.

• We can also update the run list of a node through the Chef management console. The Chef Management Console is a GUI interface through which we can manipulate the Chef run list. But it should be noted that as per the latest updates on Chef this interface is soon to be depreciated, and replaced with Chef Automate.

Unit tests are easy to run simpler tests used to test a single unit or component of the infrastructure. Chef uses ChefSpec, an in-built testing framework to write unit tests for the infrastructure. In the infrastructure point of view unit tests are verified on the resource collection created during the Chef Client run and not by actually configuring the node.

For eg: If we want to write a unit test to verify if the “Apache” package is installed, no actual installation of the Apache server is tested but just verifies if the package “Apache” is added to the resource collection list. Resource collection will have a list of all actions that will be performed in the actual Chef Client run.

Integration tests in Chef's perspective are system tests where tests are performed by provisioning a new system and configuring the system to our requirements. This takes a lot of time as we have to perform the actual configuration of the provisioned system. System tests are also a costly affair compared to unit tests as we need to provision a new system exclusively for testing purposes.

“file” resources in Chef are used to manage files in a Chef node during a chef-client run . A sample file resource is as follows:

```file ‘/tmp/index.html’ do
     content 'Hello World'
   end```

“file” resource has a name field that defines the resource block. In the above example ‘/tmp/index.html’ defines the name of the file which is managed by Chef with the path. Various actions performed by file resource is as follows:

• create : creation of a file
• delete : Delete a file
• touch : Changes the access time and modified time of the file
• nothing : Do no action unless notified by other resource
• create_if_missing : Create a file only if the file is not present in Chef node

“cookbook_file” resource is used to copy files from the “files” folder of the Chef cookbook to a path in Chef node during a chef-client run. Sample “cookbook_file” is specified as below:

```cookbook_file '/var/index.php' do
source 'index.php'
action :create
  end```

Here file “index.php” from “files” folder in the cookbook is copied to “index.php” file in “/var” path in Chef node. If file not present it’s created at the path.If the file is present and if the checksum of the “index.php” file matches the “index.php” file in the cookbook of Chef Server, no file transfer is performed.

“template” resources are used to dynamically generate static files from an embedded Ruby template placed in the “templates” folder of a cookbook. For eg:

```template '/etc/motd' do
     source 'motd.erb'
 end```

“/etc/motd” is the location at which a new file is created on Chef node during a chef-client run based on the template file “motd.erb” placed in the “templates” folder of the cookbook.A template file may contain Ruby statements and expressions unlike in “file” resource or “cookbook_file” resource.

Attributes define a node! In other words, it’s a characteristic of your node when it runs in a specific role under a specific environment against a specific cookbook or recipe. Hence attributes can be found in all these Chef components such as roles, environment, cookbooks and even as a separate set called “attribute files”. The different types of attributes available in the chef are:-

1. default: This is the most generic type of attribute and also the one with the lowest precedence. These are largely used in cookbooks so that nodes remain the same way as they are and function further. At the start of every Chef Client run, the default attributes are reset automatically
2. force_default: As the name suggests with the use of these attributes, the default attributes are forced to take up the new values assigned to them. The use of force_default attributes comes when we want our nodes to run in a different role or environment.force_default attributes defined in a cookbook have higher precedence than default attributes defined in a role or environment.
3. normal: Unlike default attributes, these are not reset during a new Chef Client run. They just persist in the node object.
4. override: Generally found in recipes and are used to override any default, force_default or normal attributes. These are also reset before a    Chef Client run.
5. forced_override: These attributes make sure that the override attributes defined in a cookbook are forced to take up the new values assigned to them.
6. automatic: These attributes are defined by Ohai to mainly understand the state of the node at the beginning of a Chef Client run. These have the highest precedence

In the Chef architecture, we have multiple Chef nodes managed by a Chef Server. Hence care must be taken to authenticate a Chef node with Chef Server. No stranger nodes should be able to send requests to Chef Server. So we use the “knife ssl fetch “ command to copy the self-signed SSL certificate of Chef Server to the trusted certificates directory in a respective node or a Chef Workstation. Only nodes or workstations with a valid SSL certificate could communicate with Chef Server without any errors. The URL for Chef Server is often specified in “client.rb” file in Chef node and “knife.rb” file in Chef Workstation. We can override this setting by supplying the URL of any Chef Server when the “knife ssl check” and “knife ssl fetch” command is run

“Knife ssl check” command is used to verify if the SSL certificate has a valid X.509 certificate property. “knife ssl check” and “knife ssl fetch” could be used to troubleshoot communication errors with Chef Server. This process is not valid for verifying communication with Hosted Chef Servers.

execute resources in Chef is used to execute a single command. All commands executed via execute resources are not idempotent and are specific to the environment in which it’s run. We need to use guards like “not_if”, “only_if” to make the execution of the resource idempotent. During a Chef Client run, guards property verifies the state of the Chef node. "execute" resources could be run alone or in combination with other in-built Chef resources like templates, etc. "execute" resources could be used when we need to reuse existing shell scripts for infrastructure configurations. We need to specify the command to run the shell script in "execute" resource block.

For eg:

```execute 'apache_test' do
     command '/usr/apachectl configtest'
end```

“apache_test” is the name provided for the "execute" resource block. The actual command run is “/usr/apachectl configtest”.

Execute resource could have only two actions, nothing and run. If the action specified is "run", the command provided is executed. If the action specified is nothing, the command provided in the "execute" block is prohibited from running. script resource is often confused with execute resource. Script resource is used to execute a script using an interpreter provided like bash etc

Chef nodes in your infrastructure may perform different roles at different times. For eg: Few of the Chef nodes would be web servers and one would be a load balancer. Grouping of the nodes into web servers and load balancer could be achieved in Chef using “roles” feature. The roles are defined through the role attributes. You can assign roles to your identical servers and all of them can go ahead and run the same run list mapped to the corresponding role. This helps in avoiding the process of running run lists manually on each node each time when you have many nodes performing the same function.

For configuring a new role for our node, we can make use of Knife commands. As a prerequisite, any previous roles or recipes on the run list can be removed using the “knife node run-list remove” command and then we can go ahead and add our newly added role using the “knife node run-list add role” command. The role attributes could only be defined as either ‘default’ or ‘override’ attributes. When a Chef Client run happens, the role attributes are matched against the attributes present on the node and if it takes precedence over default attributes, new settings are applied.

Bootstrap is the process by which the Chef Client is installed on the Chef nodes. Chef nodes could be physical, virtual or cloud machine. The bootstrap process can be done in two ways:

1. Using the “knife bootstrap” sub-command

The “knife bootstrap” command is issued from the Chef workstation. The command first creates an SSH connection with the node then installs the Chef Client along with all its dependencies on the node. This command also generates keys and register the node with Chef Server. In earlier versions of Chef Client, "ORGANIZATION-validator.pem" file was used to authenticate the Chef node with the Chef Server during the first Chef Client run. Starting from Chef Client 12.1, "USER.pem" file could be used to authenticate the new Chef node. This process is also known as “validatorless bootstrap”.

1.  Manual install in the node directly

We need to login to the Chef node and manually install the chef-client. Once it’s installed we need to manually set up the connection with the Chef Server.

Unattended Installs are often done on Chef nodes created in AWS Auto Scaling, AWS CloudFormation, Rackspace Auto Scale, and PXE. Chef Client is installed without SSH connection on the Chef nodes.

Security is an important feature in infrastructure management, in Chef this could be achieved through data bags. While preparing recipes for setting the different nodes we might need to provide some sensitive data as well for example like authorization credentials. It is not a good practice to hard code them as this may lead to many security risks. It is here where we can make use of data bags. Data bags are places where we can save these important data, the data gets stored in JSON format (Key-value pairs).

Data bags are indexed hence can be called whenever required by the recipes or cookbooks through a search. The data bags are encrypted and hence the contents of the data bags can be accessed only by having a decrypting key, this is how data bags give security to the chef infrastructure. Other than encryption strategies, a Chef Data Bag can be secured using a ‘Chef Vault’. Data bags can be created through the commands in the knife utility tool or manually. Data bags could be considered as a method to store global data shared across nodes.

Setting up environments is an important part of infrastructure management. We know that for the successful development of any product we need different environments like a development environment, testing environment, production environment, etc. The concept of environments in Chef is the extrapolation of the same. We at times want our Chef nodes to converge to a testing environment or a production environment configurations. Hence we can have a particular configuration for a testing environment and different configuration for the production environment. This switching of node behaviour is achieved through Chef environment settings. Environments help administrators to easily identify the stage where the particular node is a part of.

If we don’t declare any environments the Chef assumes the nodes to be in “_default” environment. Minimum one environment exists in an organisation. Environment attributes have higher precedence than default attributes of a node. Environment data could be stored in Ruby format or JSON format. An environment could be created as follows:

• Using Chef Management console at Chef Server
• Using the knife tool
• Creating a ruby file in an environments subfolder of chef-repo
• Creating a JSON file in chef-repo
• Using Chef Infra Server REST API

Chef Software hosts a repository for general-purpose community cookbooks called Chef Supermarket. Hence Chef users can use these designed and tested cookbooks wherever possible which in turn will reduce the infrastructure development time. But often we come across situations where we need to tweak the community cookbooks for our requirement i.e where wrapper cookbooks are created. Wrapper cookbooks are similar to normal cookbooks but they use recipes from other cookbooks also.One another scenario where wrapper cookbooks are created is where you have a large number of baseline cookbooks and we need to organise them for maintainability.

We create wrapper cookbooks and put in external recipes from other cookbooks for our use. This could be achieved by modifying

1. metadata.rb file to include details about all dependent cookbooks. All dependent cookbooks are specified in “depends” directive
2. Using ‘include_recipe‘ directive in a recipe to include external recipes from other cookbooks

Wrapper classes can be further customized to your needs through some simple attribute changes. Attribute definitions and run lists of the nodes could also be version controlled using wrapper cookbooks.

chef-repo in your Chef Workstation is a directory that holds all details about cookbooks, environments, roles, and data bags.chef-repo is like source code and need to be synchronized with version control system like Git.chef-repo has following directories and files

• .chef folder is a hidden folder that stores all validation files required for authentication between Chef node, Chef Server and Chef Workstation.
• cookbooks folder to store all cookbooks used by Chef Client to configure Chef nodes. While using "knife upload" cookbooks from this folder are updated to the Chef Server
• data_bags folder to store all data bags created for an organisation. A single data bag creates a subfolder with a JSON file.
• environments folder is used to store all details about various environments available in the Chef Server.
• roles folder contain all definitions of various roles available in the Chef Server.
• Chefignore file is used to ignore unwanted files like swap files, version control data, build output data so that these details don't get uploaded to Chef Server. Wild cards like ?*,** could be used to specify ignore patterns in chefignore files. This file could be located on any sub-folders in chef-repo.