cupcakearmy/blaze
1
0
Fork 0
mirror of https://github.com/cupcakearmy/blaze.git synced 2024-12-22 08:16:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add service worker for caching system

Browse Source
This commit is contained in:
Danny Spina 2023-06-23 16:16:35 +02:00
parent 9c50cb2c0d
commit 413e88b5cc
8 changed files with 280 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
README.md
View File

@ -1,42 +1,89 @@
# Blaze - Ultrafast Search Engine for Minimalist Browsing
<h1 style="text-align: center">BLA⚡️E</h1>
Blaze is an ultrafast search engine designed to minimize data transfer between clients and servers, enabling users to browse the web in a minimalist manner. This repository contains the core code for the Blaze search engine and lightweight page rendering.
[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/Naereen/StrapDown.js/graphs/commit-activity)
## Features
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
- **Minimalist Search**: Blaze aims to reduce the transferred data between clients and servers, providing a minimalist search experience.
- **Lightweight Page Rendering**: When a search result is clicked, Blaze generates a lightweight version of the web page using Readability library, allowing for quick loading even under challenging connection conditions.
- **Open Source**: The code for Blaze is open source and available on GitHub. You can host your own version of Blaze or contribute to its development.
[![Website shields.io](https://img.shields.io/website-up-down-green-red/https/blaze.cyclic.app)](https://blaze.cyclic.app/)
## How It Works
[![TypeScript](https://badgen.net/badge/icon/typescript?icon=typescript&label)](https://typescriptlang.org)
Blaze utilizes the following technologies and frameworks:
Blaze is an ultrafast search engine designed to minimize data transfer between clients and servers, enabling users to browse the web in a minimalist manner.
<br>
<br>
The main goal and purpose of Blaze is to allow users to search for information in all connectivity conditions, including poor, slow, or unstable connections. It also works offline, providing access to previously visited pages.
- **Express**: A fast, unopinionated web framework for Node.js, used to handle server-side requests and responses.
- **@mozilla/readability**: A library for extracting the main content of a web page, enabling the generation of lightweight pages for search results.
- **JSDOM**: A JavaScript implementation of the W3C DOM, used to parse and manipulate HTML documents in a virtual environment.
- **Got**: A lightweight HTTP client for making requests to external APIs.
## 💪 Features
The core functionality of Blaze can be summarized as follows:
- **Surf the web in all conditions**: Blaze allows users to navigate the web even in poor connectivity conditions, such as unstable or slow connections.
- **Minimalist Search**: Blaze aims to reduce data transferred between clients and servers, providing a minimalist search experience.
- **Lightweight Page Rendering**: When a search result is clicked, Blaze generates a lightweight version of the web page using the Readability library, ensuring quick loading even under challenging connection conditions.
- **Cache-based and Offline Mode**: Every page visited through Blaze is saved in the service worker cache, making it accessible offline. Additionally, the service worker provides fast access to already visited pages even when online.
1. The user initiates a search query through the Blaze web interface.
2. The query is sent to the Brave Search API to retrieve search results.
3. The received results are processed to generate a minimalist HTML page with clickable links.
4. When a search result link is clicked, the corresponding web page is fetched.
5. The fetched web page is parsed using Readability to extract the main content.
6. The extracted content is displayed as a lightweight page to the user.
## 🔧 How It Works
## Implications and Benefits
The idea behind Blaze is straightforward:
Blaze addresses the issue of slow internet connections and enables seamless browsing even in challenging conditions. By minimizing the transferred data, Blaze offers several benefits:
```mermaid
graph LR
client(Client)--GET-->blaze(Blaze)
blaze--GET-->remote(Remote Server)
remote--heavy response-->blaze
blaze--ultralight response-->client
```
Blaze acts as a middleware between the client and the servers where the requested pages are located.
Let's explore the behavior in case of no connection at all:
```mermaid
graph LR
client(Client)
blaze(Blaze)
remote(Remote Server)
sw(Service Worker)
client--GET-->sw
sw--'offline but cached or page cached'-->client
sw--'page not cached'-->blaze
blaze--GET-->remote
remote--heavy response-->blaze
blaze--ultralight response-->sw
```
As shown in the diagram, the service worker plays a central role in Blaze. The service worker's cache enables offline access to previously visited pages.
However, what happens if the client is online and requests a cached page that has been updated? In this case, the service worker first checks if the requested page is the same (in terms of HTML) as the cached page.
To achieve this, Blaze sets an HTTP header called 'X-Blaze-Etag' when parsing the page. This header is used to compare the cached page's header with the requested page's header. If they differ, the page is requested again and saved in the cache:
```mermaid
graph TB
client(Client)
blaze(Blaze)
remote(Remote Server)
sw(Service Worker)
blaze--'GET'-->remote
remote--'response'-->blaze
blaze--'parsed page with Etag'-->sw
sw--'cached page'-->client
sw--'different Etag'-->blaze
```
## 👌 Implications and Benefits
Blaze addresses the issue of slow internet connections and enables seamless browsing even in challenging conditions. By minimizing data transfer, Blaze offers several benefits:
1. **Faster Browsing**: With Blaze, users can search for information and load pages extremely quickly, even with poor internet connections.
2. **Reduced Bandwidth Consumption**: Blaze significantly reduces the amount of data transferred between clients and servers, resulting in lower bandwidth consumption.
3. **Battery and Processor Efficiency**: Browsing lightweight and minimalist web pages puts less stress on the phone's battery and processor, leading to improved device efficiency.
3. **Battery and Processor Efficiency**: Browsing lightweight and minimalist web pages puts less stress on the device's battery and processor, leading to improved efficiency.
4. **Ad-Free Experience**: Blaze pages are mostly free from ads, enhancing the browsing experience and reducing distractions.
5. **Environmental Impact**: Blaze's minimal data transfer approach can contribute to environmental sustainability by reducing battery drain, decreasing the need for frequent phone charging, and potentially lowering CO2 emissions.
5. \*\*Environmental
## Getting Started
Impact**: Blaze's minimal data transfer approach contributes to environmental sustainability by reducing battery drain, decreasing the need for frequent charging, and potentially lowering CO2 emissions. 6. **Emergency Search Engine\*\*: Blaze can be considered an "Emergency search engine." It provides access to critical information even in situations with poor connectivity.
## 🧑‍💻 Getting Started
To run your own instance of Blaze or contribute to the project, follow these steps:
@ -50,7 +97,7 @@ To run your own instance of Blaze or contribute to the project, follow these ste
Please note that you need to obtain a Brave Search key to use Blaze effectively. Visit the [Brave Search website](https://search.brave.com) for more information.
## Contributing
## ✌️Contributing
Contributions to Blaze are welcome! Feel free to open issues or submit pull requests on the GitHub repository. Please follow the existing code style and provide clear descriptions of your changes.

12
dist/index.html vendored
View File

@ -8,6 +8,7 @@
name="description"
content="Blaze: The search engine for seamless browsing in challenging connections. Discover fast results and access online content efficiently."
/>
<meta http-equiv="Cache-control" content="public" />
<title>Blaze</title>
<style>
body {
@ -62,6 +63,17 @@
t.addEventListener("click", () => {
location.href = e + "?q=" + encodeURIComponent(c.value);
});
if ("serviceWorker" in navigator) {
navigator.serviceWorker
.register("/service-worker.js")
.catch((error) => {
console.log(
"Something went wrong with installation of the service worker:",
error
);
});
}
</script>
</body>
</html>

82
dist/info.html vendored
View File

@ -1 +1,81 @@
<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><link rel="icon" type="image/x-icon" href="/favicon.svg"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="description" content="Discover Blaze, the search engine for seamless browsing in any connection condition. Minimize bandwidth with minimalist page rendering. Enhance your browsing experience."><title>Blaze - info</title><style>body{font-family:sans-serif}</style></head><body><h2>What is this site?</h2><p>Blaze is an ultrafast search engine. The purpose of Blaze is to minimize the transferred data between clients and servers, allowing users to navigate the web in a minimalist manner.</p><h2>Why?</h2><p>One day, I exceeded my monthly data limit for high-speed browsing. As a consequence, my internet speed was significantly reduced. Consequently, web pages took an extremely long time to load, and I encountered timeout errors 90% of the time. It was at that moment that I contemplated a solution to this issue and conceived the idea of Blaze. Now, I can surf the web super fast and access the information I need even without a high-speed connection.</p><h2>How?</h2><p>It's actually very straightforward. When you click on "BLAZE IT!", the URL changes to https://blaze.cyclic.app?q= followed by the query you searched for. This address is where the backend is hosted. When the site is accessed with the "q" parameter, a simple Node.js application retrieves the search results and generates an extremely lightweight version of SERP (Search Engine Results Page). From there, if you click on a link, it will be parsed and a page will be generated with only the content (I used the same library that Firefox uses to generate the reader mode). This makes it possible to navigate using only a few kilobytes of data instead of megabytes.</p><h2>Does it work?</h2><p>Yes, even with a poor internet connection, you can search for things and load pages extremely quickly. When you click on "BLAZE IT!" and select a search result, you receive a webpage that is only a few kilobytes in size instead of several megabytes. This makes it possible to load the page even under challenging connection conditions, effectively preventing timeout errors from occurring.</p><h2>Other implications</h2><p>This project was born to address an issue with slow internet connections while allowing continuous web surfing. During development, I discovered several other positive implications. Browsing a lightweight and minimalistic web puts less stress on the phone's battery and processor. It is mostly free from ads, and it likely has other, albeit possibly minor, impacts on the environment (e.g., reduced battery drain, decreased need for frequent phone charging, and lower CO2 emissions). Moreover, Blaze can serve as an "emergency search engine" for browsing the web in challenging connection situations, and it can even be a lifesaver at times.</p><h2>Open Source</h2><p><a href="https://github.com/daaanny90/blaze-this-page">The code is open source</a>, and you can host your hown version of Blaze, or contribute to it.</p><h2>Who I am?</h2><p>I am Danny, you can learn more about me <a href="https://dannyspina.com">on my website</a></p></body></html>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/x-icon" href="/favicon.svg" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta
name="description"
content="Discover Blaze, the search engine for seamless browsing in any connection condition. Minimize bandwidth with minimalist page rendering. Enhance your browsing experience."
/>
<title>Blaze - info</title>
<style>
body {
font-family: sans-serif;
}
</style>
</head>
<body>
<h2>What is this site?</h2>
<p>
Blaze is an ultrafast search engine. The purpose of Blaze is to minimize
the transferred data between clients and servers, allowing users to
navigate the web in a minimalist manner.
</p>
<h2>Why?</h2>
<p>
One day, I exceeded my monthly data limit for high-speed browsing. As a
consequence, my internet speed was significantly reduced. Consequently,
web pages took an extremely long time to load, and I encountered timeout
errors 90% of the time. It was at that moment that I contemplated a
solution to this issue and conceived the idea of Blaze. Now, I can surf
the web super fast and access the information I need even without a
high-speed connection.
</p>
<h2>How?</h2>
<p>
It's actually very straightforward. When you click on "BLAZE IT!", the URL
changes to https://blaze.cyclic.app?q= followed by the query you searched
for. This address is where the backend is hosted. When the site is
accessed with the "q" parameter, a simple Node.js application retrieves
the search results and generates an extremely lightweight version of SERP
(Search Engine Results Page). From there, if you click on a link, it will
be parsed and a page will be generated with only the content (I used the
same library that Firefox uses to generate the reader mode). This makes it
possible to navigate using only a few kilobytes of data instead of
megabytes.
</p>
<h2>Does it work?</h2>
<p>
Yes, even with a poor internet connection, you can search for things and
load pages extremely quickly. When you click on "BLAZE IT!" and select a
search result, you receive a webpage that is only a few kilobytes in size
instead of several megabytes. This makes it possible to load the page even
under challenging connection conditions, effectively preventing timeout
errors from occurring.
</p>
<h2>Other implications</h2>
<p>
This project was born to address an issue with slow internet connections
while allowing continuous web surfing. During development, I discovered
several other positive implications. Browsing a lightweight and
minimalistic web puts less stress on the phone's battery and processor. It
is mostly free from ads, and it likely has other, albeit possibly minor,
impacts on the environment (e.g., reduced battery drain, decreased need
for frequent phone charging, and lower CO2 emissions). Moreover, Blaze can
serve as an "emergency search engine" for browsing the web in challenging
connection situations, and it can even be a lifesaver at times.
</p>
<h2>Open Source</h2>
<p>
<a href="https://github.com/daaanny90/blaze-this-page"
>The code is open source</a
>, and you can host your hown version of Blaze, or contribute to it.
</p>
<h2>Who I am?</h2>
<p>
I am Danny, you can learn more about me
<a href="https://dannyspina.com">on my website</a>
</p>
</body>
</html>

31
index.ts
View File

@ -14,8 +14,9 @@ import {
blazeUrl,
injectBlazeToPageLinks,
} from "./utils.js";
import etag from "etag";
import compression from "compression";
import fs from "fs";
const app = express();
const port = 8888;
@ -36,14 +37,21 @@ const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
// Middlewares
app.use(compression());
app.use((req, res, next) => {
res.set("Cache-Control", "public, max-age=60000");
res.set("Service-Worker-Allowed", "/");
next();
});
// Routes
app.get("/", async (req, res) => {
const searchEngine = "https://api.search.brave.com/res/v1/web/search";
const query = req.query.q as string;
if (!query) {
return res.sendFile(path.join(__dirname, "/dist/index.html"));
return res.sendFile(path.join(__dirname, "/index.html"));
}
const key = process.env.CYCLIC_BRAVE_KEY;
@ -112,7 +120,7 @@ app.get("/", async (req, res) => {
${results.join("")}
<script>
${blazeFunctionality}
blazeFunctionality('${blazeUrl}')
blazeFunctionality('${blazeUrl}')
</script>
</body>
</html>
@ -120,6 +128,7 @@ app.get("/", async (req, res) => {
const minifiedSerp = minify(html, minifierOptions);
res.set("X-Blaze-Etag", etag(minifiedSerp));
res.send(minifiedSerp);
};
xhr.send();
@ -142,7 +151,7 @@ app.get("/blazed", async (req, res) => {
}
if (xhr.status === 404) {
res.sendFile(path.join(__dirname, "/dist/404.html"));
res.sendFile(path.join(__dirname, "/404.html"));
return;
}
@ -265,11 +274,21 @@ app.get("/blazed", async (req, res) => {
});
app.get("/info", (_, res) => {
res.sendFile(path.join(__dirname + "/dist/info.html"));
let Etag;
fs.readFile(path.join(__dirname + "/info.html"), "utf8", (err, data) => {
if (err) {
console.error(err);
return;
}
Etag = etag(data);
res.set("X-Blaze-Etag", Etag);
res.sendFile(path.join(__dirname + "/info.html"));
});
});
app.get("/ooops", (_, res) => {
res.sendFile(path.join(__dirname + "/dist/info_not_blazed.html"));
res.sendFile(path.join(__dirname + "/info_not_blazed.html"));
});
app.get("/favicon.svg", (_, res) => {

20
package-lock.json generated
View File

@ -12,6 +12,7 @@
"@mozilla/readability": "^0.4.4",
"compression": "^1.7.4",
"dotenv": "^16.3.1",
"etag": "^1.8.1",
"express": "^4.18.2",
"fetch": "^1.1.0",
"got": "^13.0.0",
@ -23,6 +24,7 @@
},
"devDependencies": {
"@types/compression": "^1.7.2",
"@types/etag": "^1.8.1",
"@types/express": "^4.17.17",
"@types/html-minifier": "^4.0.2",
"@types/jsdom": "^21.1.1",
@ -112,6 +114,15 @@
"@types/node": "*"
}
},
"node_modules/@types/etag": {
"version": "1.8.1",
"resolved": "https://registry.npmjs.org/@types/etag/-/etag-1.8.1.tgz",
"integrity": "sha512-bsKkeSqN7HYyYntFRAmzcwx/dKW4Wa+KVMTInANlI72PWLQmOpZu96j0OqHZGArW4VQwCmJPteQlXaUDeOB0WQ==",
"dev": true,
"dependencies": {
"@types/node": "*"
}
},
"node_modules/@types/express": {
"version": "4.17.17",
"resolved": "https://registry.npmjs.org/@types/express/-/express-4.17.17.tgz",
@ -2222,6 +2233,15 @@
"@types/node": "*"
}
},
"@types/etag": {
"version": "1.8.1",
"resolved": "https://registry.npmjs.org/@types/etag/-/etag-1.8.1.tgz",
"integrity": "sha512-bsKkeSqN7HYyYntFRAmzcwx/dKW4Wa+KVMTInANlI72PWLQmOpZu96j0OqHZGArW4VQwCmJPteQlXaUDeOB0WQ==",
"dev": true,
"requires": {
"@types/node": "*"
}
},
"@types/express": {
"version": "4.17.17",
"resolved": "https://registry.npmjs.org/@types/express/-/express-4.17.17.tgz",

8
package.json
View File

@ -2,12 +2,12 @@
"name": "blaze-this-page",
"version": "1.0.0",
"description": "The service to navigate the web also with bad connectivity",
"main": "index.js",
"main": "index.ts",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node index.js",
"start": "node ./dist/index.js",
"build": "npm run html-minify && npx tsc",
"dev": "concurrently \"npx tsc --watch\" \"nodemon -q index.js\"",
"dev": "concurrently \"npx tsc --watch\" \"nodemon -q ./dist/index.js\"",
"html-minify": "html-minifier --input-dir dist --output-dir dist --file-ext html --remove-comments --collapse-whitespace --minify-js true --minify-css true"
},
"author": "Danny Spina",
@ -16,6 +16,7 @@
"@mozilla/readability": "^0.4.4",
"compression": "^1.7.4",
"dotenv": "^16.3.1",
"etag": "^1.8.1",
"express": "^4.18.2",
"fetch": "^1.1.0",
"got": "^13.0.0",
@ -27,6 +28,7 @@
},
"devDependencies": {
"@types/compression": "^1.7.2",
"@types/etag": "^1.8.1",
"@types/express": "^4.17.17",
"@types/html-minifier": "^4.0.2",
"@types/jsdom": "^21.1.1",

62
service-worker.ts Normal file
View File

@ -0,0 +1,62 @@
/// <reference no-default-lib="true"/>
/// <reference lib="esnext" />
/// <reference lib="WebWorker" />
interface Window extends ServiceWorkerGlobalScope {}
const sw: ServiceWorkerGlobalScope = self as any;
sw.addEventListener("install", (event: ExtendableEvent) => {
event.waitUntil(
caches.open("blaze").then((cache) => {
return cache.addAll(["/"]);
})
);
});
// TODO: I had to use any, the solutions above (references) were ineffective
// (but un the install works).
// must be fixed.
self.addEventListener("fetch", (event: any) => {
event.respondWith(
fetch(event.request)
.then((networkResponse) => {
if (networkResponse.status === 200) {
const currentEtag = networkResponse.headers.get("X-Blaze-Etag");
return caches.open("blaze").then((cache) => {
return cache.match(event.request).then((cachedResponse) => {
const cachedEtag = cachedResponse
? cachedResponse.headers.get("X-Blaze-Etag")
: null;
if (currentEtag === cachedEtag) {
return cachedResponse;
} else {
return cache
.put(event.request, networkResponse.clone())
.then(() => {
return networkResponse;
});
}
});
});
} else {
return networkResponse;
}
})
.catch(() => {
return caches.open("blaze").then((cache) => {
return cache
.match(event.request)
.then((cachedResponse) => {
if (cachedResponse) {
return cachedResponse;
} else {
return new Response("You are offline");
}
})
.catch(() => {
return new Response("An error occurred");
});
});
})
);
});

5
tsconfig.json
View File

@ -55,7 +55,7 @@
// "sourceMap": true, /* Create source map files for emitted JavaScript files. */
// "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
// "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
"outDir": ".", /* Specify an output folder for all emitted files. */
"outDir": "./dist", /* Specify an output folder for all emitted files. */
// "removeComments": true, /* Disable emitting comments. */
// "noEmit": true, /* Disable emitting files from a compilation. */
// "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
@ -105,7 +105,8 @@
},
"include": [
"index.ts",
"utils.ts"
"utils.ts",
"service-worker.ts"
],
"exclude": [
"node_modules"