"Experienced Lead Developer oriented towards problem-solving and analyzing impacts, both technically and functionally, Enjoy's working with web technologies with a strong focus on quality code and robust system architecture."
- Home
- Blog
- Web Development
- Overview of Puppeteer and puppeteer-core in Node.js
HomeBlogWeb DevelopmentOverview of Puppeteer and puppeteer-core in Node.js
Share
Overview of Puppeteer and puppeteer-core in Node.js
Published
05th Sep, 2023
Views
Read TimeRead it in
15 Mins
Automating developer tools to simplify development and testing has become an integral part of web development as more teams are trying to focus on fast and efficient deliverables. Who doesn't want to automate mundane and repetitive manual tasks when it helps you become more efficient as a developer?
Here is where 'Puppeteer' comes into the picture. It is an open-source Node.js library that helps automate and simplify development by providing control over the browser Developer tools. It assists developers in writing and maintaining simple and automated tests, due to which most of the things that were done in the browser manually can now be done by using the puppeteer.
The Puppeteer website describes it as follows -
"Puppeteer is a Node library that provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default but can be configured to run full (non-headless) Chrome or Chromium."
Puppeteer is put together by the same team that was behind the making of Google Chrome the reason why it is so well maintained. It programmatically uses JavaScript to help us perform common actions on the Chromium browser, via a simple and easy-to-use API. For a comprehensive understanding of how Puppeteer works you need basic knowledge of JavaScript, ES6+, and Node.js. KnowledgeHut provides a great resource to learn Node for the same.
Getting Started with Puppeteer
Let us get started by understanding what Puppeteer exactly is before we even start to see how it's utilized. Puppeteer is nothing but a high-level API built by Google or in layman's terms it is a human-readable interface that's used to interact with the webpages through the browser (Chrome or Chromium).
It is powerful as it abstracts many nitty-gritty details we would have to deal with otherwise and therefore, makes things very easy for us. It communicates with the browser using DevTools Protocol. This is the standard protocol that is used to expose Chrome's functionality to various development tools while chrome runs in headless mode for automation.
A headless browser in this context chrome is a web browser, without a graphical user interface (GUI), which is mainly used for automated testing. As it spins up a new Chrome instance every time it's initialized, it's not always performant but it's the most precise way to automate testing with Chrome as it uses the actual browser under the hood.
Now we know what Puppeteer is, so let's check what all can be done using this powerful API. Though its use case varies, some of the notable ones are as follows:
- can be used to scrape web pages
- used for automating form submissions
- performs browser automation
- creation of server-side rendered versions of single-page apps (SPA's)
- widely used to take screenshots
- create automated testing scenarios
- widely used to generate PDFs from web pages
A great place to start with npm libraries in this Full stack course.
Installing and Running Puppeteer
As a first step, let's initialize our application with a package.json file. For this, just run the following command –
npm init
Now let's start by installing Puppeteer in our application using the below command.
npm install puppeteer
As Puppeteer installs into your application, it also downloads a recent version of Chromium by default.
Point to note, Puppeteer can also be used for Chrome browser but works best with the version of Chromium it is bundled with. We know now that Puppeteer creates an instance of the web browser and then manipulates the pages of the same, so let us see it in action. For this let's create an implementation of puppeteer for navigating to a web page, then take a screenshot of the same and save it as "example.png" to understand its working better –
We start by creating a screenshot.js file at the root of the project, and in this file, we will first need to require Puppeteer as given below:
const puppeteer = require('puppeteer'); Then we will need to call the launch method on it, which in turn will initialize the instance of Puppeteer for us.
const browser = await puppeteer.launch();
A point to remember here is that this method is asynchronous and returns a Promise. So, we will have to wait for it to get the browser instance.
Then we call newPage on it and go to KnowledgeHut and take a screenshot of it and save the screenshot as knowledge.png.
const page = await browser.newPage()
await page.goto('https://www.knowledgehut.com')
await page.screenshot({ path: 'knowledge.png' })
Then we call the browser close function, to quit the running browser.
await browser.close() The whole code in the screenshot.js file should now look like below
const puppeteer = require('puppeteer')
const main = async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://www.knowledgehut.com')
await page.screenshot({ path: 'knowledge.png' })
await browser.close()
}
main()
To run the above code in the terminal type:
node screenshot Understanding Node basics and how to run Node commands on a terminal check out this cool KnowledgeHut learn Node resources.
Default Runtime Settings
Some default runtime settings need to be noted while working with Puppeteer:
1. Uses Headless mode
Puppeteer launches Chromium in headless mode. As we have already discussed, a headless browser is a web browser without a graphical user interface (GUI). If you need to launch a full version of Chromium, it's required that you set the headless option as false when launching a browser:
const browser = await puppeteer.launch({ headless: false }); // default is true 2. Runs a bundled version of Chromium
By default, a specific version of Chromium is downloaded and used by Puppeteer, this guarantees that the API will work out of the box.
Using Puppeteer with a different version of Chrome or Chromium is also possible, just pass in the path of the executable when creating a Browser instance like below
const browser = await puppeteer.launch({executablePath: '/path/to/Chrome'}); 3. Creates a fresh user profile Puppeteer creates its browser user profile which it cleans up on every run. You can see your current running profile by going to
chrome://version
Getting Started with Headless Chrome
Let us start by discussing why it is important to have headless Chrome.
A headless browser aka, a web browser without a GUI, is a great tool for automated testing and server environments where there is no need for a visible UI. For example, running tests against a real web page, creating a PDF from it, or just inspecting how the browser renders an URL. The most effortless way to initiate the headless browser is to open the Chrome binary from the command line (Command Prompt) using some easy-to-use commands. Like For Chrome 59+, you can start chrome with the --headless flag like below:
chrome --headless --disable-gpu
Note: As of recent updates you'll also want to include the --disable-gpu flag if you're running on Windows.
Let's check out some useful command line flags to perform common tasks.
Create a PDF
chrome --headless --disable-gpu --print-to-pdf https://www.chromestatus.com/
Taking screenshots
chrome --headless --disable-gpu --screenshot https://www.chromestatus.com/
Running the --screenshot will create a file named screenshot.png in the current working directory. Headless Chrome is best suited for testing, so let's check out how to do the same.
Automated Testing with Headless Chrome
The advantage of using Headless Chrome for testing directly instead of running the tests on Node is that your JavaScript tests are carried out in the same environment as the users of the website.
Headless Chrome gives you an added advantage by providing the real browser context excluding the memory overhead of running a full version of Chrome. So let us practically check how we can do automated testing using headless chrome. For this, we will need the help of libraries like Karma, Mocha, and Chai.
Karma is a testing harness that works with all the most popular testing frameworks (Jasmine, Mocha, QUnit) and Chai is an assertion library that works with Node and the browser.
Installation
To install Karma, related plugins, and the test runners we will use the below command:
npm i --save-dev karma karma-chrome-launcher karma-mocha karma-chai npm i --save-dev mocha chai
Now we will configure Karma by creating a karma.conf.js file that uses the ChromeHeadless launcher.
karma.conf.js
module.exports = function(config) {
config.set({
frameworks: ['mocha', 'chai'],
files: ['test/**/*.js'],
reporters: ['progress'],
port: 9876, // karma web server port
colors: true,
logLevel: config.LOG_INFO,
browsers: ['ChromeHeadless'],
autopatch: false,
// singleRun: false, // Karma will capture browsers, runs the tests and exits
concurrency: Infinity
})
} Alternatively, you can Run ./Node_modules/karma/bin/karma init karma.conf.js to generate the Karma configuration file in your project.
Write a test
Create a test in /test/test.js of your project.
/test/test.js
describe('#indexOf()', () => {
it('should return -1 when the value is not present, () => {
assert.equal(-1, [1,2,3].indexOf(4));
});
});
}); Now we add a test script in package.json that runs Karma with the mentioned settings.
package.json
"scripts": {
"test": "karma start --single-run --browsers ChromeHeadless karma.conf.js"
} When you run your tests (npm test), Headless Chrome will start up and generate the output with a similar result to the terminal:
yarn test ve.24.6 $ karma start --single-run --browsers MyHeadlessChrome karma.conf.js 13 06 2017 10:49:44.947:INFO [karma]: Karma v1.7.0 server started at http://8.0.0.8:9876/ 13 06 2017 10:49:44.958:INFO [launcher]: Launching browser MyHeadlessChrome with unlimited concurrency 13 06 2017 10:49:44.957:INFO [launcher]: Starting browser Chrome Headless 13 06 2017 10:49:45.269:INFO [HeadlessChrome 0.0.0 (Mac OS X 10.12.0)]: Connected on socket xAbkPZP9UYENOMRAAAA with id 86428182 HeadlessChrome 0.0.(Mac OS X 10.12.4): Executed 1 of 1 SUCCESS (0.008 secs / 0.001 secs) Done in 1.60s.
Debugging Tips
There are multiple options to debug using Puppeteer, in this section we will walk through some of them.
1. Turn off headless mode
Sometimes it's useful to launch a full version of the browser to see what the browser is displaying instead of launching in headless mode. In such cases, you can turn off headless mode by using the flag "headless: false".
const browser = await puppeteer.launch({headless: false}); 2. Slow it down
We can also slow down Puppeteer operations by the specified number of milliseconds using the "slowMo" option. It's another way to see what is going on step by step or to check how the browser responds to each of your actions.
const browser = await puppeteer.launch({ headless: false, slowMo: 350, // slow down by 350ms }); 3. Capture console output
Another handy way to debug is to listen for the console event. To listen to console events, use the below code when debugging code on page.evaluate():
page.on('console', msg => console.log('PAGE LOG:', msg.text()));
await page.evaluate(() => console.log(`url is ${location.href}`)); 4. Use debugger in application code browser
There are two kinds of execution contexts, one is the Node.js which is running the test code, and the other is the browser running the application code being tested.
This allows you to debug code in the application's code browser; that is you add code inside evaluate() by using {devtools: true} when launching Puppeteer:
const browser = await puppeteer.launch({devtools: true}); 5. Add an evaluate statement with debugger inside
Add debugger to an existing evaluate statement:
await page.evaluate(() => { debugger; }); The test will stop executing in the above evaluation statement, and chromium will intercept in debug mode.
6. Use debugger in Node.js
This will let you debug the test code itself. For example, If you want to step over await page.click() you won't be able to run await page.click() in DevTools console due to this Chromium bug.
So if you want to try this out you can do this in the Node.js script and see the click happen in the application code browser.
- Add debugger; in your test file, for example:
debugger;
await page.click('a[target=_blank]'); - Set headless to false using the flag
- Run Node --inspect-brk, eg Node --inspect-brk Node_modules/.bin/jest tests
- In Chrome open chrome://inspect/#devices and click inspect
- In the newly opened test browser, type F8 to resume test execution
- Now the debugger will be hit and you can start debugging in the test browser
Troubleshooting Puppeteer
Let's explore some troubleshooting options that come with puppeteer to make life a little easier.
Chrome headless doesn't launch on Windows
At times Headless chrome can fail to launch on windows due to some chrome policies that can enforce running certain extensions on browsers' launch. The thing to understand here is that Puppeteer passes the --disable-extensions flag by default and therefore will fail to launch when such policies are enforced.
Try running without the flag as a workaround for this issue.
const browser = await puppeteer.launch({
ignoreDefaultArgs: ['--disable-extensions'],
}); Chrome is downloaded but fails to launch on Node.js 14
Sometimes you can get an error that looks like this when trying to launch Chromium:
(Node:15505) UnhandledPromiseRejectionWarning: Error: Failed to launch the browser process! spawn /Users/.../Node_modules/puppeteer/.local-chromium/mac-756035/chrome-mac/Chromium.app/Contents/MacOS/Chromium ENOENT
This indicates that the browser was downloaded but was not extracted correctly. This commonly happens due to a bug in Node.js v14.0.0 which broke extract-zip, this module is used by Puppeteer to extract and place browser downloads into the right place.
To troubleshoot this bug, please use Node.js v14.1.0 or version higher.
Code Transpilation Issues
While using a JavaScript transpiler like babel or TypeScript and calling evaluate() with an async function might throw some code Transpilations issues.
This happens because puppeteer uses Function.prototype.toString() to serialize functions while transpilers might change the output code in such a way that it becomes incompatible with Puppeteer.
A workaround to this problem is to instruct the transpiler not to mess up the code, for eg, configure TypeScript to use the latest ECMA version ("target": "es2018"). Another workaround would be to use string templates instead of functions:
await page.evaluate(`(async() => {
console.log('1');
})()`); Puppeteer – Examples
There are lots of use cases for Puppeteer, I have mentioned some examples for the same below:
To create a PDF for the given website
Check out the code below to create a PDF of a website.
const puppeteer = require('puppeteer')
const main = async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com', { waitUntil: 'networkidle2' })
await page.pdf({ path: 'hn.pdf', format: 'A4' })
await browser.close()
}
main() Note:
networkidle2 comes in handy for pages that do long polling or any other side activity and considers navigation to be finished when there are no more than two network connections for at least 500ms.
Sign In to Facebook Using Puppeteer
const puppeteer = require('puppeteer')
const SECRET_EMAIL = 'example@gmail.com'
const SECRET_PASSWORD = 'secretpass123'
const main = async () => {
const browser = await puppeteer.launch({
headless: false,
})
const page = await browser.newPage()
await page.goto('https://facebook.com', { waitUntil: 'networkidle2' })
await page.waitForSelector('#login_form')
await page.type('input#email', SECRET_EMAIL)
await page.type('input#pass', SECRET_PASSWORD)
await page.click('#loginbutton')
// await browser.close()
}
main() Conclusion
I hope going through this article gives you an insight into puppeteer and helps you familiarize yourself with the tool and its capabilities.
I think it's one of the best software to automate trivial tasks with an easy-to-use API, if you also feel so, go ahead and automate the boring tasks in your day-to-day life with Puppeteer.
Frequently Asked Questions (FAQs)
1. What is puppeteer used for?
It is an open-source library used in JavaScript based tools like NodeJS, Angular, React that helps automate and simplify development by providing control over the browser Developers tools.
Puppeteer is nothing but a high-level API built by Google, in layman's terms it is a human-readable interface that's used to interact with the webpages through the browser (Chrome or Chromium).
It is powerful as it abstracts many nitty-gritty details we would have to deal with otherwise, thus making things very easy for us.
2. Is puppeteer better than selenium?
The preference between Selenium and Puppeteer boils down to your requirements.
In case the primary focus is on testing browser applications, especially on multiple browsers, Selenium is always a better choice as it is purpose-built for cross-platform testing.
Though, If you are solely focused on JavaScript and Chrome, Puppeteer is seemingly a better fit.
3. How do I run a puppeteer in Node?
start by installing Puppeteer in our application using the below command.
npm install puppeteer
As Puppeteer installs into your application, it also downloads a recent version of Chromium by default.
Then require or import puppeteer in your project file like below:
const puppeteer = require('puppeteer'); Then we will need to call the launch method on it, which in turn will initialize the instance of Puppeteer for us.
const browser = await puppeteer.launch();
Upcoming Web Development Batches & Dates
| Name | Date | Fee | Know more |
|---|
Course Advisor