Home Running End-to-End Tests with Playwright on AWS Lambda
Post
Cancel

Running End-to-End Tests with Playwright on AWS Lambda

Posted Feb 15, 2021
By Lari Haataja
11 min read

I’ve been previously using Selenium for some of my end-to-end testing needs, but with the latest project, I wanted to check out if any of the alternatives would better fit my needs. Selenium has been the go-to tool for browser automation for ages. It’s extremely powerful but also takes quite a lot of time to master and configure. After some googling, I identified the three most potential choices to be:

Puppeteer (by Google) uses the DevTools protocol with Chromium (also offering experimental support for Firefox) making it a little less interesting option, while Playwright (by Microsoft) supports Chromium, Firefox and WebKit. The core team behind Playwright is the same that worked originally on Puppeteer. While the first two are more general browser automation frameworks, Cypress has been developed with a focus on end-to-end testing. Cypress provides an open-source test runner, supporting Chromium and Firefox, and a closed-source commercial dashboard for monitoring your test runs.

After setting up some test projects and browsing through the docs, I came to the conclusion that Playwright would provide the best browser support and the most flexibility. So I went ahead and did some more testing with it. Unfortunately, you still need some infrastructure to host your Playwright setup. You can quite easily integrate it into your CI process but maybe that’s not what you are looking for. Your use case could be something else, like monitoring a website or testing a product that doesn’t have a CI pipeline (maybe I’ll tell more about my use cases in another blog post).

Being kind of a serverless enthusiast, I wanted to figure out if Playwright could be run on AWS Lambda. In this blog post I’ll walk through how to do it step by step.

Container image as AWS Lambda function

Even though it’s possible to add extra dependencies (such as the Chromium binary) to AWS Lambda as layers, it’s somewhat difficult to maintain. Fortunately, Amazon just launched a new feature in December (2020) that lets you package and deploy Lambda functions as container images — a game changer for running Playwright on AWS Lambda!

So I started to look into building a container image with all the Playwright dependencies and deploying that as a Lambda function.

The code

The final version of the code presented in this blog post can be found from my GitHub repo https://github.com/lari/playwright-aws-lambda-example (v0.0.1).

The project directory looks like this:

1
2
3
4
5
6

playwright-aws-lambda-example/
- function/
| - index.js
| - package.json
- Dockerfile
- entrypoint.sh

Requirements

The function

I created a simple Node.js function that uses Playwright to open the Google search front page and print the page title to the console. You can define the browser (webkit, chromium or firefox) as an event parameter called browser.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

const { webkit, chromium, firefox } = require('playwright');

const browserTypes = {
    'webkit': webkit,
    'chromium': chromium,
    'firefox': firefox,
};

exports.handler = async (event, context) => {
    let browserName = event.browser || 'chromium';
    let browser;
    try {
        browser = await browserTypes[browserName].launch({});
        const context = await browser.newContext();
        const page = await context.newPage();
        await page.goto('http://google.com/');
        console.log(`Page title: ${await page.title()}`);
    } catch (error) {
        console.log(`Error ${error}`);
        throw error;
    } finally {
        if (browser) {
            await browser.close();
        }
    }
}

Remember to create package.json and add playwright as a dependency.

The Dockerfile

I started to build a docker image based on this post on the AWS blog.

Let’s use mcr.microsoft.com/playwright:focal as the base image and install the build dependencies for AWS Lambda Runtime API.

1
2
3
4
5
6
7
8
9
10
11

FROM mcr.microsoft.com/playwright:focal as build-image

# Install aws-lambda-ric build dependencies
RUN apt-get update && apt-get install -y \
    g++ \
    make \
    cmake \
    unzip \
    libcurl4-openssl-dev \
    autoconf \
    libtool

Next, we’ll copy the function directory (the actual lambda function code) to the image and install Node.js packages. We also need to install the Node.js Runtime Interface Client (RIC).

1
2
3
4
5
6
7
8
9
10
11
12

# Define custom function directory
ARG FUNCTION_DIR="/function"

# Create function dir and install node packages
RUN mkdir -p ${FUNCTION_DIR}
COPY function/ ${FUNCTION_DIR}

WORKDIR ${FUNCTION_DIR}

RUN npm install

RUN npm install aws-lambda-ric

Finally, we’ll add Lambda Runtime Interface Emulator (RIE) to help us with local testing. RIE lets you invoke your function on a locally running container through HTTP requests.

1
2
3
4
5
6
7
8
9

# Add Lambda Runtime Interface Emulator and use a script in the ENTRYPOINT for simpler local runs
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/local/bin/aws-lambda-rie
RUN chmod 755 /usr/local/bin/aws-lambda-rie
COPY entrypoint.sh /
RUN chmod 755 /entrypoint.sh
ENTRYPOINT [ "/entrypoint.sh" ]

# Set function handler
CMD ["index.handler"]

Entrypoint

The entrypoint.sh script checks for the existence of AWS_LAMBDA_RUNTIME_API environment variable. If it’s found (i.e. running on a real AWS Lambda environment), the script starts RIC normally, and otherwise it starts RIC through the emulator. The CMD instruction (function handler) will be passed to ENTRYPOINT (runtime interface client) as an argument.

1
2
3
4
5
6

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then 
  exec /usr/local/bin/aws-lambda-rie /usr/bin/npx aws-lambda-ric $1
else
  exec /usr/bin/npx aws-lambda-ric $1
fi

Running your code

Locally with Docker

Build docker container image:

1

docker build -t playwright-aws-lambda-example:latest .

Run the container image and map your port 9000 to port 8080 of the container:

1

docker run -p 9000:8080 playwright-aws-lambda-example:latest

Invoke the function with an HTTP request. This is possible because we added the Runtime Interface Emulator:

1

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Deploying to AWS Lambda

In the following examples, replace YOUR_AWS_ACCOUNT_ID with your AWS Account Id.

Create a repository on AWS ECR (Elastic Container Registry):

1

aws ecr create-repository --repository-name playwright-aws-lambda-example --image-scanning-configuration scanOnPush=true

Tag docker image for ECR repository:

1

docker tag playwright-aws-lambda-example:latest YOUR_AWS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/playwright-aws-lambda-example:latest

Login:

1

aws ecr get-login-password | docker login --username AWS --password-stdin YOUR_AWS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com

Push image to ECR:

1

docker push YOUR_AWS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/playwright-aws-lambda-example

Finally, you need to create an AWS Lambda function and deploy it with the container image you just uploaded. This can be done in the AWS web console.

Too good to be true? Let there be issues…

Everything seems to work fine on your local docker container but after deploying your hard work to AWS Lambda you’ll notice how things are never so simple. Next, I’ll go through the issues I came across on the AWS Lambda environment and explain how they’re solved.

Browser executable path

The function will fail on the actual AWS Lambda environment with an error message:

1
2

Failed to launch webkit because executable doesn't exist at 
/home/sbx_user1051/.cache/ms-playwright/webkit-1402/pw_run.sh

The base image (mcr.microsoft.com/playwright:focal) installs browser executables under the home directory of a user called pwuser and creates symbolic links for other users. However, AWS Lambda uses its own pool of users called sbx_userXXXX (sbx probably stands for sandbox) that don’t exist when the image is built. Thus, the users won’t have symbolic links to the browser executables under their own home directories.

Luckily Playwright has a custom path option for the browser executable in the browserType.launch() method. So I created a helper function that takes the expected browser executable path and returns a path under the /home/pwuser/ directory:

1
2
3
4

getCustomExecutablePath = (expectedPath) => {
    const suffix = expectedPath.split('/.cache/ms-playwright/')[1];
    return  `/home/pwuser/.cache/ms-playwright/${suffix}`;
}

You can then use this for the executablePath launch option:

1
2
3

browser = await browserTypes[browserName].launch({
    executablePath: getCustomExecutablePath(browserTypes[browserName].executablePath()),
});

WebKit just works

After providing the correct executable path, the WebKit browser seems to work out of the box. No issues at all! (So far. There might be issues with some more special use cases).

No support for multiprocessing (?)

When running the function with Chromium, it fails at a process called “start_thread”. I’m not an expert on this, but I’ve understood that the function fails when Chromium is trying to spawn a new thread/process for something, probably GPU related.

Here’s part of the stack trace:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

[err] #0 0x5588292260b3 content::internal::ChildProcessLauncherHelper::PostLaunchOnLauncherThread()
[err] #1 0x558829225b4c content::internal::ChildProcessLauncherHelper::StartLaunchOnClientThread()
[err] #2 0x55882976e952 content::VizProcessTransportFactory::ConnectHostFrameSinkManager()
[err] #3 0x55882b45541b mojo::SimpleWatcher::Context::Notify()
[err] #4 0x55882976e952 content::VizProcessTransportFactory::ConnectHostFrameSinkManager()
[err] Task trace buffer limit hit, update PendingTask::kTaskBacktraceLength to increase.
[err] 
[err] Received signal 6
[err] #0 0x55882ad34919 base::debug::CollectStackTrace()
[err] #1 0x55882aca56d3 base::debug::StackTrace::StackTrace()
[err] #2 0x55882ad344fb base::debug::(anonymous namespace)::StackDumpSignalHandler()
[err] #3 0x7f5bba2093c0 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so+0x153bf)
[err] #4 0x7f5bb8a6e18b gsignal
[err] #5 0x7f5bb8a4d859 abort
[err] #6 0x55882ad33495 base::debug::BreakDebugger()
[err] #7 0x55882acb6d6d logging::LogMessage::~LogMessage()
[err] #8 0x5588293372f7 content::(anonymous namespace)::IntentionallyCrashBrowserForUnusableGpuProcess()
[err] #9 0x55882933504e content::GpuDataManagerImplPrivate::FallBackToNextGpuMode()
[err] #10 0x5588293339e3 content::GpuDataManagerImpl::FallBackToNextGpuMode()
[err] #11 0x55882933e11f content::GpuProcessHost::RecordProcessCrash()
[err] #12 0x5588291c46c3 content::BrowserChildProcessHostImpl::OnProcessLaunchFailed()
[err] #13 0x558829226250 content::internal::ChildProcessLauncherHelper::PostLaunchOnClientThread()
[err] #14 0x558829226495 base::internal::Invoker<>::RunOnce()
[err] #15 0x55882acf57a6 base::TaskAnnotator::RunTask()
[err] #16 0x55882ad06e13 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
[err] #17 0x55882ad06aea base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
[err] #18 0x55882ad5af47 base::MessagePumpLibevent::Run()
[err] #19 0x55882ad0769b base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
[err] #20 0x55882acdd9cd base::RunLoop::Run()
[err] #21 0x5588291e1ec8 content::BrowserProcessSubThread::IOThreadRun()
[err] #22 0x55882ad1f014 base::Thread::ThreadMain()
[err] #23 0x55882ad450bf base::(anonymous namespace)::ThreadFunc()
[err] #24 0x7f5bba1fd609 start_thread
[err] #25 0x7f5bb8b4a293 clone
[err]   r8: 0000000000000000  r9: 00007f5bb440ed30 r10: 0000000000000008 r11: 0000000000000246
[err]  r12: 00007f5bb4410058 r13: 00007f5bb4410050 r14: 00007f5bb4410040 r15: 00007f5bb440f800
[err]   di: 0000000000000002  si: 00007f5bb440ed30  bp: 00007f5bb440ef80  bx: 00007f5bb4411700
[err]   dx: 0000000000000000  ax: 0000000000000000  cx: 00007f5bb8a6e18b  sp: 00007f5bb440ed30
[err]   ip: 00007f5bb8a6e18b efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
[err]  trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[err] [end of stack trace]
[err] Calling _exit(1). Core file will not be generated.

AWS Lambda runs in an environment that has several limitations. One of them seems to be the lack of multiprocessing support. Modern browsers spawn multiple processes with a couple of different strategies. For e.g. rendering is often done with its own process to not block other things if page rendering is slow or jams. Some browsers also have separate processes for each tab. You can read more about Process Models for Chromium here.

Luckily with Chromium, you can use a launch flag called --single-process to disable the use of multiple processes. “In this model, both the browser and rendering engine are run within a single OS process.” Again, the browserType.launch() method has an option called args: “Additional arguments to pass to the browser instance”.

It’s worth noticing what the docs say about this --single-process flag: “It is not a safe or robust architecture, as any renderer crash will cause the loss of the entire browser process. It is designed for testing and development purposes, and it may contain bugs that are not present in the other architectures.”

Let’s add a constant with launch arguments for each browser type:

1
2
3
4
5
6
7

const browserLaunchArgs = {
    'webkit': [],
    'chromium': [
        '--single-process',
    ],
    'firefox': [],
}

Then launch the browser with the args option:

1
2
3
4

browser = await browserTypes[browserName].launch({
    executablePath: getCustomExecutablePath(browserTypes[browserName].executablePath()),
    args: browserLaunchArgs[browserName],
});

Firefox… not solved yet

With Firefox you get another cryptic stack trace that looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

cloned child 31
[err] ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[err] ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[err] Unable to init server: Could not connect: Connection refused
<process did exit: exitCode=null, signal=SIGSEGV>
=========================== logs ===========================
<launching> /home/pwuser/.cache/ms-playwright/firefox-1221/firefox/firefox -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-xmDdON -juggler-pipe -silent
<launched> pid=20
[err] *** You are running in headless mode.
[err] ExceptionHandler::GenerateDump cloned child 31
[err] ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[err] ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[err] Unable to init server: Could not connect: Connection refused
<process did exit: exitCode=null, signal=SIGSEGV>
============================================================
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.Error
    at /function/node_modules/playwright/lib/server/firefox/ffConnection.js:54:63
    at new Promise (<anonymous>)
    at FFConnection.send (/function/node_modules/playwright/lib/server/firefox/ffConnection.js:53:16)
    at Function.connect (/function/node_modules/playwright/lib/server/firefox/ffBrowser.js:44:24)
    at Firefox._connectToTransport (/function/node_modules/playwright/lib/server/firefox/firefox.js:28:38)
    at Firefox._innerLaunch (/function/node_modules/playwright/lib/server/browserType.js:90:36)
    at async ProgressController.run (/function/node_modules/playwright/lib/server/progress.js:85:28)
    at async Firefox.launch (/function/node_modules/playwright/lib/server/browserType.js:55:25)
    at async BrowserTypeDispatcher.launch (/function/node_modules/playwright/lib/dispatchers/browserTypeDispatcher.js:30:25)
    at async DispatcherConnection.dispatch (/function/node_modules/playwright/lib/dispatchers/dispatcher.js:147:28)

With my limited knowledge, I’ve reasoned that this is the same multiprocess issue as with Chromium. Unfortunately, Firefox doesn’t have a similar launch argument to force single processing. Thus, I’ve not been able to solve this yet and I’m afraid it might not be even possible.

Conclusions

In the end, I managed to get two out of three browsers working on AWS Lambda: WebKit and Chromium. It’s of course better than nothing, but it seems that AWS Lambda is a tough environment to run modern web browsers. However, I’m planning to continue testing and see if I face more issues with proper real-life use cases.

All the code can be found in my GitHub repo playwright-aws-lambda-example. The first version (tagged: ‘v0.0.1’) is almost identical to the code examples shown in this blog post.

I hope you enjoyed reading my post and found it useful. I’m happy to hear any feedback and especially ideas on how to make Firefox work with AWS Lambda.

Playwright
node.js aws lambda docker testing e2e testing
This post is licensed under CC BY-SA 4.0 by the author.
Contents