Search

Monitoring AWS Machine using Azure Log Analytics - Cloud Computing

What is Azure Log Analytics?Logs make your life easier and help everyone in the organization understand what you do and what's Actually going on with the software. Ensure that you follow best practices of Azure Log Analytics for monitoring AWS machines effectively.Logs are created by network devices, applications, operating systems, and programmable or smart devices. They comprise several messages that are chronologically arranged and stored on a disk, in files, or in an application like a log collector.Firstly, we will try to understand why we need logs to monitor AWS services.We need logs because, with Azure Log Analytics, you can gather and search log data from all your resources, whether they reside on Azure, on another cloud host, or on-premises. Then you can transform the gathered data into rich analytics with AI-enhanced insights into your environment.  How Log Analytics worksThe log data from Virtual machines and other cloud resources, capture via some Agent which we need to install on VMs and Resource can be named as Connected Sources which further will be gathered in records and sent to OMS repository that is the part of Azure Cloud resource and stored in the Azure cloud.After receiving logs to log analytics it further is available for log search and other activity like Set Alerts, Dashboard View, Power BI View, and Export into Excel and Data files. Data collection in Azure Log AnalyticsData can be collected in azure Log Analytics in different ways –Agents on Windows and Linux virtual machines send telemetry from the guest operating system and applications to Log Analytics according to Data Sources that you configure.Connect a System Center Operations Manager management group to Log Analytics to collect data from its agents.Azure services such as Application Insights and Azure Security Center store their data directly in Log Analytics without any configuration.Write data from PowerShell command line or Azure Automation runbook using Log Analytics cmdlets.If you have custom requirements, then you can use the HTTP Data Collector API to write data to Log Analytics from any REST API client.Azure Monitor                   Data sources for events and performanceVirtual Machines               Data sources for events and performanceOperations Manager         Data from management group agentsApplication Insights           Application requests and exceptionsAzure Security Center       Security eventsPowerShell                        PowerShell command line or runbookData Collector API            Rest API for custom dataIncoming data automatically indexed. Data types and tables automatically created.Log AnalyticsData available through log search and smart analytics to multiple channels.Design and test queries and analyze data         AnalyticsVisualize data in Azure portal                            DashboardsWorkflows consuming Log Analytics data         Logic AppsAutomatically respond critical conditions           AlertsExport for visualization with other sources        Power BIPowerShell command line or runbook               PowerShellRest API for custom application                         Log Search APIWork flow of Log AnalyticsNow let's find out the workflow of log analytics like how it collects data, analyze, visualize and alerts.So first look into Collect part–The logs can be collected via: -Event LogsCustom App LogsIIS LogsCrash DumpsPerformance DataWe can also filter the type of logs by just checking and unchecking the log types.Once the collection has done we need to Analyse the data which can be done by: -Filter based on attributesAnalyze data with Kusto Query languageSort dataExport log data to Excel and Power BIConditional FilteringAfter that, we can Visualize all the logs in an attractive DashboardAlerts can be configured on the basis of event conditions like once the value will be reached greater than or less than the limit threshold it will automatically generate the alerts and take configured actions like sending main, sending a message and start runbook (part of azure automation).Steps to collect Data and log from AWS MachinesAWS VMs could be of Windows and Linux, so we can install the agent which can be download from azure log analytics page by just selecting the type of the OS and the Bit of the VMs.After installing the Agent we need to configure the agent by entering the workspace id and key which will be provided by the azure after configuring. We can get the all the logs within some Hours in the portal.Supported AWS OS and VersionsLet’s discuss the OS and versions of VMs that are supported on AWS Cloud.If you are using Windows, the configuration should be:Windows Server 2008 Service Pack 1 (SP1) or laterWindows 7 SP1 and laterIf you are using Linux, the configuration should be:Amazon Linux 2012.09 to 2015.09 (x86/x64)CentOS Linux 5, 6, and 7 (x86/x64)Oracle Linux 5, 6, and 7 (x86/x64)Red Hat Enterprise Linux Server 5, 6 and 7 (x86/x64)Debian GNU/Linux 6, 7, and 8 (x86/x64)Ubuntu 12.04 LTS, 14.04 LTS, 16.04 LTS (x86/x64)SUSE Linux Enterprise Server 11 and 12 (x86/x64)After verifying the supported configuration, we can successfully install the agent and receive the logs.Kusto to query AWS Machine LogsKusto is a log analytics cloud platform optimized for ad-hoc big data queries.Kusto Query Reference Portal: https://docs.loganalytics.ioThe Kusto Query Language is used to query Azure services.OMS (Operational Management Suite)The Microsoft Operations Management Suite (OMS), previously known as Azure Operational Insights, is a software as a service platform that allows an administrator to manage on-premises and cloud IT assets from one console.Azure OMS provides 4 types of services: -Log Analytics: Monitor and analyze the availability and performance of different resources including physical and virtual machines.Automation: Automate manual processes and enforce configurations for physical and virtual machines.Backup: Backup and restore critical data.Site Recovery: Provide high availability for critical applications.Management SolutionsManagement solutions leverage services in Azure provides additional insight into the operation of a particular application or service.Here, we can manually select the required solution and add to the home page where we can only see the logs of that solution type.ReportsThe final reports of logs can be exported in Excel and Power BI that can be displayed in table and chart format.Conclusion:Azure Log Analytics is a very powerful tool to capture different types of system log. Kusto Query plays a very important role in extracting insights from the log file. Also, custom reports can be prepared using KUSTO Query,which helps the organization in saving many man-hours.Detailed reports and easy to export to excel and power helps to keep the troubleshooting and diagnosis handy.Reports can be embedded in any website with live refresh data and code snipped can be generated within power BI.Choose the right Azure logging service for AWS monitoring and use it to save many man-hours and reduce the time of troubleshooting and diagnosis.
Monitoring AWS Machine using Azure Log Analytics - Cloud Computing
Raju
Rated 4.0/5 based on 2 customer reviews
Raju

Raju Kumar

Blog Author

Certified Azure Solution Architect and MCT, also specialized in AWS & Google Cloud Platform.

Having 6+ Years of Enterprise Product Development, worked in 4 MNC's across global teams.
I am a regular contributor at various technical conferences, Meetups & Community events to help the community to spread cloud awareness.
Things I do: Training, Consulting and Product Development. Let's Connect for a discussion !!

Posts by Raju Kumar

Monitoring AWS Machine using Azure Log Analytics - Cloud Computing

What is Azure Log Analytics?Logs make your life easier and help everyone in the organization understand what you do and what's Actually going on with the software. Ensure that you follow best practices of Azure Log Analytics for monitoring AWS machines effectively.Logs are created by network devices, applications, operating systems, and programmable or smart devices. They comprise several messages that are chronologically arranged and stored on a disk, in files, or in an application like a log collector.Firstly, we will try to understand why we need logs to monitor AWS services.We need logs because, with Azure Log Analytics, you can gather and search log data from all your resources, whether they reside on Azure, on another cloud host, or on-premises. Then you can transform the gathered data into rich analytics with AI-enhanced insights into your environment.  How Log Analytics worksThe log data from Virtual machines and other cloud resources, capture via some Agent which we need to install on VMs and Resource can be named as Connected Sources which further will be gathered in records and sent to OMS repository that is the part of Azure Cloud resource and stored in the Azure cloud.After receiving logs to log analytics it further is available for log search and other activity like Set Alerts, Dashboard View, Power BI View, and Export into Excel and Data files. Data collection in Azure Log AnalyticsData can be collected in azure Log Analytics in different ways –Agents on Windows and Linux virtual machines send telemetry from the guest operating system and applications to Log Analytics according to Data Sources that you configure.Connect a System Center Operations Manager management group to Log Analytics to collect data from its agents.Azure services such as Application Insights and Azure Security Center store their data directly in Log Analytics without any configuration.Write data from PowerShell command line or Azure Automation runbook using Log Analytics cmdlets.If you have custom requirements, then you can use the HTTP Data Collector API to write data to Log Analytics from any REST API client.Azure Monitor                   Data sources for events and performanceVirtual Machines               Data sources for events and performanceOperations Manager         Data from management group agentsApplication Insights           Application requests and exceptionsAzure Security Center       Security eventsPowerShell                        PowerShell command line or runbookData Collector API            Rest API for custom dataIncoming data automatically indexed. Data types and tables automatically created.Log AnalyticsData available through log search and smart analytics to multiple channels.Design and test queries and analyze data         AnalyticsVisualize data in Azure portal                            DashboardsWorkflows consuming Log Analytics data         Logic AppsAutomatically respond critical conditions           AlertsExport for visualization with other sources        Power BIPowerShell command line or runbook               PowerShellRest API for custom application                         Log Search APIWork flow of Log AnalyticsNow let's find out the workflow of log analytics like how it collects data, analyze, visualize and alerts.So first look into Collect part–The logs can be collected via: -Event LogsCustom App LogsIIS LogsCrash DumpsPerformance DataWe can also filter the type of logs by just checking and unchecking the log types.Once the collection has done we need to Analyse the data which can be done by: -Filter based on attributesAnalyze data with Kusto Query languageSort dataExport log data to Excel and Power BIConditional FilteringAfter that, we can Visualize all the logs in an attractive DashboardAlerts can be configured on the basis of event conditions like once the value will be reached greater than or less than the limit threshold it will automatically generate the alerts and take configured actions like sending main, sending a message and start runbook (part of azure automation).Steps to collect Data and log from AWS MachinesAWS VMs could be of Windows and Linux, so we can install the agent which can be download from azure log analytics page by just selecting the type of the OS and the Bit of the VMs.After installing the Agent we need to configure the agent by entering the workspace id and key which will be provided by the azure after configuring. We can get the all the logs within some Hours in the portal.Supported AWS OS and VersionsLet’s discuss the OS and versions of VMs that are supported on AWS Cloud.If you are using Windows, the configuration should be:Windows Server 2008 Service Pack 1 (SP1) or laterWindows 7 SP1 and laterIf you are using Linux, the configuration should be:Amazon Linux 2012.09 to 2015.09 (x86/x64)CentOS Linux 5, 6, and 7 (x86/x64)Oracle Linux 5, 6, and 7 (x86/x64)Red Hat Enterprise Linux Server 5, 6 and 7 (x86/x64)Debian GNU/Linux 6, 7, and 8 (x86/x64)Ubuntu 12.04 LTS, 14.04 LTS, 16.04 LTS (x86/x64)SUSE Linux Enterprise Server 11 and 12 (x86/x64)After verifying the supported configuration, we can successfully install the agent and receive the logs.Kusto to query AWS Machine LogsKusto is a log analytics cloud platform optimized for ad-hoc big data queries.Kusto Query Reference Portal: https://docs.loganalytics.ioThe Kusto Query Language is used to query Azure services.OMS (Operational Management Suite)The Microsoft Operations Management Suite (OMS), previously known as Azure Operational Insights, is a software as a service platform that allows an administrator to manage on-premises and cloud IT assets from one console.Azure OMS provides 4 types of services: -Log Analytics: Monitor and analyze the availability and performance of different resources including physical and virtual machines.Automation: Automate manual processes and enforce configurations for physical and virtual machines.Backup: Backup and restore critical data.Site Recovery: Provide high availability for critical applications.Management SolutionsManagement solutions leverage services in Azure provides additional insight into the operation of a particular application or service.Here, we can manually select the required solution and add to the home page where we can only see the logs of that solution type.ReportsThe final reports of logs can be exported in Excel and Power BI that can be displayed in table and chart format.Conclusion:Azure Log Analytics is a very powerful tool to capture different types of system log. Kusto Query plays a very important role in extracting insights from the log file. Also, custom reports can be prepared using KUSTO Query,which helps the organization in saving many man-hours.Detailed reports and easy to export to excel and power helps to keep the troubleshooting and diagnosis handy.Reports can be embedded in any website with live refresh data and code snipped can be generated within power BI.Choose the right Azure logging service for AWS monitoring and use it to save many man-hours and reduce the time of troubleshooting and diagnosis.
Rated 4.0/5 based on 2 customer reviews
Monitoring AWS Machine using Azure Log Analytics -...

What is Azure Log Analytics?Logs make your life ea... Read More

How To Save Up To 40% On Azure Bill Without Buying Any Cost Management Software

I have seen many projects get shut down because of the Azure usage cost. Being a senior developer, I was asked to look over the azure usage and optimize the usage to reduce cost. In this article, we will discuss various ways for effective Azure cloud architecture cost optimization that can easily save up to 40% on Azure cost without purchasing any Cost Management Software. I'll also share some deep insights, which IT Managers doesn't care about, and which has a huge impact on Azure Bill.Different cost management system:1.CloudynCloudyn is an Azure cost management software that provides features like Report on cost and usage, Categorize by resource tags, Create and manage cost and usage budgets, Create alerts on cost and usage budgets, Eliminate idle cloud resources, Right-size cloud resources, Chargeback features including cost markup, redistribution, and custom charges, Customize recommendation thresholds and Categorize costs with custom meta-tags2. ProXcioProXcio contains features like cost analytics, usage analytics, filtered table views, exec dashboards, best practice reports, tag-based analytics, tag-based budgeting, budget analytics, aggregated reporting, limits data stored, limit range for analytics, CSV data import, EA Account drill down, multi-users support, no credit card required, support, cost forecasting, e-mail alerts, data export, Reports scheduling, API access, azure list price invoicing3. RackNapRackNap contains features like White-Label Marketplace, Subscription Billing, Support Helpdesk, Customer Self-service Portal, CRM, Sales and Marketing, Business Intelligence, Own Catalog Management, Partner Management, Inventory Management, Core Customizations, 3rd Party Integrations, Online Training, Admin Users, Deployment Countries, Deployment Model, Services for Launch.4. Azure CostsAll Azure Plans (incl. EA), Unlimited data retention Includes all Professional Features, Multi-Contract & User Support, Team & Enterprise Support, Subscription Based Permissions, Branding & Customizable Widgets, Azure Active Directory Support, Data Access via RESTful APIPrice Comparison:The table below compares the prices of different Azure cost management software available:Review Azure usage and costsCost Analysis Vs Cost AllocationDrill into various data segments itemized from the billing file using Cost Analysis Reports. The reports enable granular cost navigation across Azure raw billing data.After you create a cost model, Cost Allocation reports are available. It matches the data to the usage and tag data of the Azure Account.Cost Over Time:Cost Over Time report displays spends over time to allow you to observe trends and detect irregularities in your deployment. It includes main cost contributors such including ongoing costs and one-time RI fees are being spent during a selected time frame.Use Actual Cost Over Time to see cost trends over time and find irregularities in costAmortized Cost:Amortized cost is that accumulated portion of the recorded cost of a fixed asset that has been charged to expense through either depreciation or amortization. Amortized Cost Reports shows non-usage-based service fees or one-time payable costs and spread their cost over time evenly during their lifespan. For example, one-time fees might include:Reserved Instances purchase feesAzure Marketplace itemsAnnual security component feesThis report displays the main cost contributors within a specified time range and includes ongoing usage costs and one-time RI fees, amortized over the term of the asset or reservation.Custom Charges:Enterprise and CSP often provide additional services to their customers along with their own cloud consumption. You can define these customs charges for added service and additional discounts if any. The list of customs charges doesn't show the different rates that you may be charging.5 things to consider saving to save costIf you are using a public cloud like Azure, it is crucial to know the ways to save cost as your bill is based on consumption. Below are the 5 tips explained to lower Azure pricing and optimize hosting costs:1. Select Azure Reserved VM instancesWhat is RI (Reserved Instance)?An Azure Reserved Virtual Machine Instance (RI) is a virtual machine (VM) on the Microsoft Azure public cloud that has been reserved for dedicated use on a one- or three-year basis.RIs require a one-time, upfront payment and offer customers a discount of up to 72% when compared to Microsoft's standard on-demand, pay-per-use VM pricing model.Azure shows 3 options for Discount -Pay as you go -  You only need to pay for how much you will use.1 year reserved (~29% savings) – 29% discount will be given for reserving VMs for 1-year use.3 years reserved (~43% savings) --43% discount will be given for reserving VMs for 1-year use.2. Compare cost before choosing datacenter regionThe different VM pricing tiers do vary in price from region to region.https://azure.microsoft.com/en-in/pricing/calculator/It combines the pricing data for all VM instance sizes across all Azure regions.Estimate your expected monthly bill using our Pricing Calculator and track your actual account usage and bill at any time using the billing portal. Set up automatic email billing alerts to be notified if your spend goes above an amount you configure.3. Make use of Azure Hybrid BenefitThe most cost-effective cloud for your Windows Server or SQL Server migration which helps you-Save up to 80% on Windows Server with Azure Hybrid Benefit and Reserved InstancesSave up to 55%1 on migrations to Azure SQL Database2 with Azure Hybrid BenefitGo at your own pace - move a few workloads or entire data centers)Maximise your investment in Microsoft Server SoftwareNow is the time to move to Azure and reap the rewards of cloud technology, including the ability to scale up or down quickly, pay only for what you use and save on compute power. Whether you are deploying new virtual machines, moving a few workloads or migrating your data centers as part of your hybrid cloud strategy, the Azure Hybrid Benefit provides big savings as you move to the cloud.4.  Use different load calculator to identify the required loadThere are few additional Database resources whose price cannot be calculated based on days and storage. It needs to specify throughput, DTU, and many additional add-ons.Two types of calculators for databases-1) SQL Database Calculatorhttps://dtucalculator.azurewebsites.net/If you are a developer using SQL Server, you've probably heard of Azure SQL Database and you've probably been thinking about migrating your on-premise or VM-based SQL Server database(s) to Azure SQL Database. If so, you've probably asked yourself, "which service tier and performance level should I use and how many database throughput units (DTUs) am I using now?" This calculator will help you determine the number of DTUs for your existing SQL Server database(s) as well as a recommendation of the minimum performance level and service tier that you need before you migrate to Azure SQL Database. Knowing the minimum service tier will allow you to get the performance you need while minimizing your costs.2) Cosmos Db Calculatorhttps://www.documentdb.com/capacityplannerTo help customers fine-tune their Azure Cosmos DB throughput estimations, there is a web-based tool to help estimate the request unit requirements for typical operations, including document creates, reads, and deletes.5. Make use of BYOL (Bring Your Own License)What is BYOLBYOL, or “bring your own license,” is the process you can use to deploy software that you already have a license. When you BYOL, you are responsible for managing your own licenses. You are responsible for managing true-ups and renewals as required under your Volume Licensing agreement. In addition, you must submit a new verification form when you renew your agreement and when you deploy any previously unverified products.How to go for BYOL?As a customer using License Mobility through Software Assurance, you must complete a license verification process. Microsoft will verify the eligible license with active Software Assurance and send a confirmation once the verification process is completed.Azure now have to Bring Your Own License (BYOL) images of Windows Server and Windows 10 directly in the marketplace.This is what you needed to do before:Install Windows 10 or Windows Server on an On-Premise machineSysprep the installationUpload the vhd to a storage accountCreate a VM (by template or script) using the custom imageThis is what you need to do now to achieve the same thing:Create a VM (by template or script) using the new marketplace BYOL imageHope you found this article helpful to reduce and optimize your Azure costs. Understand and manage your Azure spend effectively with the help of above 5 cost optimization strategies for Azure bill.
Rated 4.0/5 based on 2 customer reviews
How To Save Up To 40% On Azure Bill Without Buying...

I have seen many projects get shut down because of... Read More