How to: Chrome extension (manifest v3) - capture network logs using webRequest API
🟪

How to: Chrome extension (manifest v3) - capture network logs using webRequest API

Date
Sep 27, 2022
Tags
React
Chrome Extension
Front-end
TypeScript

Introduction

One of the first tasks I was assigned to at my new workplace ( ✨Deep BlueDot ✨) was to find a way to fetch console logs (info, errors, warnings, asserts etc) and HAR network logs(Network requests and responses) of a webpage and access it through chrome extension.
Visit Devign here:
Devign provides the essential context and data for developers such as console logs, network logs, dev environment, user flow, user footprints, and more—to resolve the issue.
This article shines light upon how the Devign chrome extension fetches the network log information (HAR - HTTP Archive) using Chrome’s webRequest API which is useful for debugging.

⚠️ Warning

HAR logs capture all network information including personal information such as cookies or passwords. Be careful about whom you share the log information with!

How the Chrome Extension webRequest API works

Manifest

Your manifest.json must include webRequest permission in order to access webRequests API
notion image

Side Note

The web request API presents an abstraction of the network stack to the extension.
Internally, one URL request can be split into several HTTP requests (for example to fetch individual byte ranges from a large file) or can be handled by the network stack without communicating with the network.
notion image

Background (in version 3, referred to as service workers)

What service workers do: listen for and respond to events in order to enhance the end user's experience.

webRequest API events used:

onBeforeSendHeaders

Fired before sending an HTTP request, once the request headers are available. This may occur after a TCP connection is made to the server, but before any HTTP data is sent.
chrome.webRequest.onBeforeSendHeaders.addListener( callback: function, filter: RequestFilter, extraInfoSpec?: OnBeforeSendHeadersOptions[], )

onHeadersReceived

Fired when HTTP response headers of a request have been received.
chrome.webRequest.onHeadersReceived.addListener( callback: function, filter: RequestFilter, extraInfoSpec?: OnHeadersReceivedOptions[], )

src/background/index.ts

Note: The following example stores the network log data in an array but Devign stores the data in a queue.
  • Ignoring requests initiated by the extension with :if(details.initiator?.includes('chrome-extension://')) return;
  • Storing requestTimeStamp and responseTimeStamp separately
  • Example of the Request, Response Header structures (Variable - depends on the individual request type)
Request Headers
notion image
Response Headers
notion image
notion image

src/background/api.ts - Post the collected network log data to backend server

notion image

Content Scripts

This is the part of the extension which can access the Document Object Model (DOM) of the webpage and tells the service workers when to start collecting the network log data.
Devign extension uses a toggle button to decide whether to collect logs or not. When toggled on, the content script sends a message to the background to capture the network logs using the following code:
notion image

Here’s what it looks like in the Dashboard

Once the Network log data has been stored in the backend server, it can be parsed and displayed as shown below.
notion image
notion image

References: