license / docs pass:

- add new docs for loading, exploring, sharing, embedding, and offline usage
- add license file / license banner

electron app:
- for now, don't bundle docs, open in native browser

embeds:
- support 'embed' option, 'replayonly' or 'full' (default if no url)

ui:
- display package.json version in info menu

package.json:
- update scripts to match README
- move electron entrypoint into extraMetadata
- add scripts for packaging as module

README: minor edits
This commit is contained in:
Ilya Kreymer
2020-06-11 00:42:10 -07:00
parent 948b232fbf
commit b9a4089c6b
24 changed files with 556 additions and 126 deletions
-79
View File
@@ -1,79 +0,0 @@
---
layout: default
title: Browsing Web Archives
nav_order: 1
parent: Usage
permalink: /docs/browsing
---
# Browsing Web Archives via the ReplayWeb.page
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
## Loading Web Archives
Using [https://replayweb.page](https://replayweb.page), you can select a web archive file from a local
file or remote URL to load into the site.
Archives loaded by ReplayWeb.page are identified by their unique URL, which can be entered directly or selected from a local file browser.
### Remote Archives
To load a remote archive, simply enter the URL of the archive and click `Replay!`.
(See: [Supported Locations](locations.md) for details on where archives can be loaded from).
The archive will be downloaded, either fully or [only as needed (if possible)](streaming-archives.md) and presented on the archive page.
The system supports WARC files, as well as several other formats (See: [Supported Formats](formats.md) for more details on supported formats)
Once loaded, the page is redirected to the archive collection page.
### Local Archives
Using the `Choose File` button, you can pick any local archive file to load for replay.
The click `Replay` to load the file.
The archive is laoded in the browser directly and *not* uploaded anywhere -- it never leaves your computer!
(This process is similar to how [Webrecorder Player](https://github.com/webrecorder/webrecorder-player) works, but happens entirely in the browser.
It will be identified by a `file://` URL and is not accessible outside your browser.
## Archive View
Once an archive is loaded, you are taken to the archive view, and you are ready to explore and replay the web archive!
This view has several tabs:
- **Curated Pages** - This tab exists if the archive has curated lists of pages. Currently this is available only for Webrecorder WARCs, but may be possible
with other formats (see: [Web Archive Zip Format](webarchive-zip-format.md)
- **URL Resources** - This tab allows searching the archive by URLs, as well as by MIME type. For many archives with no page or curatorial metadata, this is
a way to explore the archive data in more detail.
Once a URL is selected, you will be taken to the Replay tab which renders the archived page directly in your browser.
## Index of Archives
The ReplayWeb.page homepage lists an index of all archives loaded in your browser.
Once loaded, the archive remains cached in the browser for quick access. This view will be unique to your browser.
To remove the archive from your browser, click on the 'X'. (Of course, this does not delete the archive from its original location, only your local copy of it)
<hr>
Next: [Sharing Links to Archived Pages](linking.md)
+82
View File
@@ -0,0 +1,82 @@
---
layout: default
title: Embedding ReplayWeb.page
nav_order: 2
permalink: /docs/embedding
---
## Embedding Web Archives with ReplayWeb.page
A key goal of ReplayWeb.page is to make embedding web archives into other sites as easily as possible,
as easy as it is to embed PDFs, for example.
ReplayWeb.page provides the `<replay-web-page>` HTML tag to support embedding.
Embedding requires just two parts, loading the frontend ui and the backend service worker.
For example, to embed a web archive stored on s3 at `s3://webrecorder-builds/warcs/netpreserve-twitter.warc`
(shorthand for: `https://webrecorder-builds.s3.amazonaws.com/warcs/netpreserve-twitter.warc`), you can first add
the following snippet to your HTML page:
{: .bg-blue-000 .text-grey-lt-000 .cap-header}
my-web-archive-embed.html
```html
<script src="https://unpkg.com/replaywebpage@1.0.0/ui.js"></script>
<replay-web-page source="s3://webrecorder-builds/warcs/netpreserve-twitter.warc"
url="https://twitter.com/netpreserve"></replay-web-page>
```
This will load the frontend UI.
Since ReplayWeb.page requires a service worker, it is necessary to add a service worker path
from where the web archive will be served. Create a subdirectory (eg. `replay/`) and place the following
one-line script
{: .bg-blue-000 .text-grey-lt-000 .cap-header}
./replay/sw.js
```javascript
importScripts("https://unpkg.com/replaywebpage@1.0.0/sw.js");
```
Thus, if the HTML snippet was added to `https://my-site.example.com/path/my-web-archive-embed.html`
then the sw.js should be added such that it is at: `https://my-site.example.com/path/replay/sw.js`.
That's it! Loading `https://my-site.example.com/path/my-web-archive-embed.html` should now load the web arhive.
(Be sure to add sizes to the `<replay-web-page>` tag as needed to size the embed).
You can replace `s3://webrecorder-builds/warcs/netpreserve-twitter.warc` with any web archive hosted on your site,
eg. `https://my-site.example.com/warcs/my-warc-file.warc`.
{: .fs-3 .pad .bg-grey-lt-100}
If the file is loaded from a different origin, your site must have CORS access to download the web archive.
### Versioning
Note that the above example uses the paths as:
- `https://unpkg.com/replaywebpage@1.0.0/ui.js`
- `https://unpkg.com/replaywebpage@1.0.0/sw.js`
Another alternative would be:
- `https://cdn.jsdelivr.net/npm/replaywebpage@1.0.0/ui.js`
- `https://cdn.jsdelivr.net/npm/replaywebpage@1.0.0/sw.js`
These URLs point to a specific version of ReplayWeb.page software released on NPM, eg. `1.0.0`, meaning that your replay should stay stable, even if ReplayWeb.page is updated.
You can choose another of ReplayWeb.page (or even try different versions) to ensure that you have tbest available replay.
This addresses the potential issue of older sites breaking when web archive replay software is updated.
For production use, it is advised against linking to the latest version, eg. `https://replayweb.page/ui.js`
and `https://replayweb.page/sw.js` as these will be updated frequently and make break your embed.
+66
View File
@@ -0,0 +1,66 @@
---
layout: default
title: Exploring Web Archives
nav_order: 2
parent: Usage
permalink: /docs/exploring
---
## Exploring Archives (Browse, Search and Replay)
The ReplayWeb.page homepage lists an index of all loaded archives. You can also search by archive title or source
and filter by date loaded or title.
Once loaded, the archive remains cached in the browser for quick access. This view will be unique to your browser.
To remove the archive from your browser, click on the 'X'. (Of course, this does not delete the archive from its original location, only your local copy of it)
### Archive Views
Although primarily designed for replay, ReplayWeb.page also offers several ways to interact with web archives.
The archive view presents several tabs:
- **Story** - This Story view presents lists of curated pages, as developed by the creator of the web archive.
This option is only shown if there is a curated story. As curated lists are not a standard part of WARC, only WARCs exported from Webrecorder.io/Conifer can have this option.
The new [Web Archive Collection (WACZ)](web-archive-collection-format) can also include curated lists.
- **Pages** - The Pages view presents all pages in the web archive. As pages are not a standard part of WARC format,
generally only WARCs from Webrecorder.io/Conifer will have pages.
The new [Web Archive Collection (WACZ)](web-archive-collection-format) can also store pages.
- **Page Resources** - This view allows searching the archive by URLs, as well as by common MIME type.
For many archives with no page or curatorial metadata, this is a way to explore the archive data in more detail.
This view is available for all archives that only store raw data.
- **Replay** - The view presents a replay of the archived web content in a mini-browser directly your browser. The view allows entering a URL directly. Clicking on links on any of the other views will switch to the **Replay** view.
### Search
Both the Page Resources view provide ways to search the archive directly.
#### Searching Page Resources
Page Resources allows searching by URL only, with additional sorting options.
Searches can be done by exact url, by url prefix, or by any string contained in the URL.
The URL Prefix option is best for searching large archives that require on-demand loading.
The contains option will not find any URLs that have not yet been loaded.
#### Searching Pages
The Page view search includes page titles, urls and page full text search, if available.
ReplayWeb.page will currently generate full ext search data from WARC pages automatically.
ReplayWeb.page will soon load existing extracted full-text data as well.
<hr>
Next: [Sharing Links to Archived Pages](sharing.md)
@@ -1,7 +1,7 @@
---
layout: default
title: 'Supported Formats'
nav_order: 2
nav_order: 3
description: 'Supported Formats'
permalink: /docs/formats
parent: Reference
@@ -9,18 +9,20 @@ parent: Reference
## Supported Formats
ReplayWeb.Page supports the following archive formats.
Format is currently determined based on the extension, though more extensive detection may be added later.
ReplayWeb.Page supports the archive formats listed below.
Format is currently determined based on the file extension.
The `.wacz` refers to the newly proposed [Web Archive Collection Zip Format](web-archive-collection).
| Format | Extensions | Status |
|:--------|:--------------------|:--------------|
| WARC | `.warc`, `.warc.gz` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| HAR | `.har` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| HAR | `.har` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress |
| WBN | `.wbn` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> Experimental |
| ARC | `.arc` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-red-000"> Not Supported |
| CDX | `.cdx`, `.cdxj` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| ZIP | `.zip`, `.waz` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress |
| WACZ | `.wacz` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress |
-8
View File
@@ -1,8 +0,0 @@
---
layout: default
title: Linking to Web Archives
nav_order: 2
parent: Usage
permalink: /docs/linking
---
+92
View File
@@ -0,0 +1,92 @@
---
layout: default
title: Loading Web Archives
nav_order: 1
parent: Usage
permalink: /docs/loading
---
# Browsing Web Archives via the ReplayWeb.page
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
## Loading Web Archives
Using [https://replayweb.page](https://replayweb.page), you can select a web archive file from a local
file or remote URL to load into the site.
Archives loaded by ReplayWeb.page are identified by their unique URL, which can be entered directly or selected from a local file chooser.
### Loading Local Web Archives
Using the `Choose File...` button, you can pick any local archive file to load for replay.
The click `Load` to load the file.
The archive is laoded in the browser directly and *not* uploaded anywhere -- it never leaves your computer!
The archive will be identified by a `file://` URL and is not accessible outside your browser. As expected,
this URL is not shareable.
{: .fs-3 .pad .bg-grey-lt-100}
This behavior is similar to how [Webrecorder Player](https://github.com/webrecorder/webrecorder-player) worked, but happens entirely in the browser. <br>The [ReplayWeb.page App](https://github.com/webrecorder/replayweb.page/releases) is also an option for [opening local files when offline](offline#replaywebpage-as-standalone-desktop-electron-app)
### Loading Remote Web Archives
To load a remote archive, simply enter the URL of the archive and click `Load`.
{: .fs-3 .pad .bg-grey-lt-100}
See [Supported Locations](locations) for details on where archives can be loaded from.
The archive will be downloaded, either fully or [only as needed (if possible)](streaming-archives.md) and presented on the archive page.
The system supports WARC files, as well as several other formats
{: .fs-3 .pad .bg-grey-lt-100}
See: [Supported Formats](formats) for more details on supported formats.
### Loading Remote Archives from Google Drive
ReplayWeb.page also supports loading archives directly from Google Drive.
ReplayWeb.page can be used to open .warc, .warc.gz files directly from google drive using the following steps.
1. Install [ReplayWeb.page Google Drive Integration](https://gsuite.google.com/u/2/marketplace/app/replaywebpage/160798412227)
2. Navigate to any WARC (.warc, .warc.gz) files or WACZ (Web archive collection) files that you have on google drive.
3. You should see an 'Open With ReplayWeb.page' option at the top.
4. Clicking on this should redirect to ReplayWeb.page and begin loading. Google Drive loaded archives will have a source URL starting with `googledrive://`.
### Loading Large Files: Full vs On-Demand Loading
Due to the nature of the WARC format, the entire file must be read on first use to generate an index.
For WARC files **>25MB**, only the index is initially stored in the browser, and the actual content is loaded 'on-demand',
when the content is first accessed. This leads to faster loading and saves memory when dealing with large archives.
[Web Archive Collection (WACZ)](web-archive-collection-format) are always loaded on-demand, as no indexing is required.
The initial archive view should load almost instantly as a result.
If an archive could not be loaded, an error will be displayed instead of the progress.
Please report any issues encountered via [GitHub Issues](https://github.com/webrecorder/replayweb.page/issues)
<hr>
Once an archive has successfully loaded, you will be redirected to the archive view from where you can explore the archive.
Next: [Exploring the Archive](explore)
@@ -1,7 +1,7 @@
---
layout: default
title: 'Supported Locations'
nav_order: 1
nav_order: 2
description: 'Supported Locations'
parent: Reference
permalink: /docs/locations
@@ -24,9 +24,9 @@ The following is a list of currently supported URL schemes and details below:
| `http://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| `file://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| `s3://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| `googledrive://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In progress |
| `ipfs://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> Partial |
| `dat://` | <span class="d-inline-block p-2 mr-1 v-align-middle text-grey-lt-000 bg-purple-000"> Planned |
| `googledrive://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
| `ipfs://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-purple-000"> Planned |
| `hyper://` | <span class="d-inline-block p-2 mr-1 v-align-middle text-grey-lt-000 bg-purple-000"> Planned |
### HTTP and HTTPS
@@ -67,11 +67,11 @@ The `ipfs://` url scheme is provide as a convenience and resolves to using the C
A better implementation with IPFS directly is also possible.
### DAT
### DAT/Hyper
{: .d-inline-block }
Help Wanted
{: .label .label-yellow }
Support for `dat://` is planned to allow true peer-to-peer transfer of web archives.
Support for `hyper://` is planned to allow true peer-to-peer transfer of web archives.
+41
View File
@@ -0,0 +1,41 @@
---
layout: default
title: Offline Use
nav_order: 4
parent: Usage
permalink: /docs/offline
---
# Offline Use
ReplayWeb.page is available for offline use in several ways.
## ReplayWeb.page Offline Cache
After ReplayWeb.page is loaded, it can automatically be used offline without requiring an internet connection.
If a connection is not available, all web archive already loaded will continue to be available.
[Loading local archives](loading#loading-local-web-archives) should fully work without an internet connection.
## ReplayWeb.page as Progress Web App
ReplayWeb.page can also be installed as [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) and be available as a standalone app on desktop and mobile.
This is still experimental, but should provide the same behavior as regular https://replayweb.page/
## ReplayWeb.page as Standalone Desktop (Electron) App
ReplayWeb.page is also available as a standalone Electron App. In this mode, ReplayWeb.page
ships its own version of Chrome.
Although the Web Page/Progressive Web App versions have almost the same features, the standalone App is useful for two primary reasons:
1) For viewing web archives that contain Flash. Flash is not supported by default in most browsers and will soon be removed. The ReplayWeb.page App will bundle a Flash plugin to continue to make Flash accessible for web archives.
2) Primary use for viewing offline web archive. If your primary use is to view web archives stored locally,
the ReplayWeb.page App is a better option, as it provides direct access to the file system.
Once loaded, ReplayWeb.page App can continue to load web archives from disk.
The desktop app fully replaces [Webrecorder Player](https://github.com/webrecorder/webrecorder-player)
When using [https://replayweb.page](https://replayweb.page), you may be asked to re-open a local file because a web site can not have persistent access to your file system.
+2 -2
View File
@@ -1,11 +1,11 @@
---
layout: default
title: Reference
nav_order: 2
nav_order: 3
has_children: true
permalink: /docs/reference
---
## Reference
A guide to the file formats, source locations and query arguments supported by ReplayWeb.Page
A guide to the file formats, specifications, source locations and other APIs to be supported by ReplayWeb.Page
+74
View File
@@ -0,0 +1,74 @@
---
layout: default
title: Sharing Web Archives
nav_order: 3
parent: Usage
permalink: /docs/sharing
---
## Sharing Links
ReplayWeb.page links are designed to be shareable with others, as long as they have access.
You can share links to any view, including Page or Page Resoures search, or Replay view.
ReplayWeb.page encodes the web archive source and view parameters in the URL query and hash string.
Simply copy the entire location url, and you can share with others!
ReplayWeb.page will parse the URL, load the source web archive, then load the view or replay from that archive.
For example, given the following URL:
```
https://replayweb.page/?source=s3://webrecorder-builds/warcs/netpreserve-twitter.warc#view=replay&url=https%3A%2F%2Ftwitter.com%2Fnetpreserve&ts=20190603053135
```
ReplayWeb.page will load the web archive from `s3://webrecorder-builds/warcs/netpreserve-twitter.warc`, switch to
the replay view via `https://twitter.com/netpreserve` at timestamp `20190603053135`
The following URL will load the same web archive, and show the search results for Pages that contain the word 'WARC':
```
https://replayweb.page/?source=s3://webrecorder-builds/warcs/netpreserve-twitter.warc#view=pages&query=WARC
```
### No Local Files
For obvious reasons, URLs that have a `source=file://...` can not be shared as they refer to a local file on your machine.
Loading that URL by anyone else will result in an error.
{: .fs-3 .pad .bg-grey-lt-100}
A future goal is to provide an option to share local web archives peer-to-peer via the DAT `hyper://` protocol.
### Sharing Google Drive Archives
An archive loaded from google drive can be shared with others, as long as they have access.
Google Drive Archive will have a `source=googledrive://...`.
Due to restrictions of Google Drive, you will likely be required to authorize with your Google Account to access an archive from Google Drive.
Public files shared from Google Drive can sometimes be loaded without requiring authorization, but this is only supported
for small files.
To share a Google Drive web archive with colleagues and friends:
1. Ensure they you have [loaded the web archive from Google Drive](loading#loading-remote-archives-from-google-drive)
2. Ensure that your friends/colleagues also have the [ReplayWeb.page Google Drive Integration](https://gsuite.google.com/u/2/marketplace/app/replaywebpage/160798412227) installed.
3. Share the file with them via Google Drive. This is the most surefire way to work.
OR.
3. You can share the direct ReplayWeb.page URL directly. This does not always work, and you may need to share the Google Drive link instead.
In this way, web archives can be shared within private networks using existing access control settings in Google Drive.
<hr>
Next: Offline Usage Options
+17 -2
View File
@@ -10,11 +10,26 @@ permalink: /docs/index.html
# ReplayWeb.Page
## Serverless / Client-Side Web Archive Replay
A browser-based system to replay archived web pages as accurately as possible, with all interactive elements preserved.
{: .fs-6 .fw-300 }
ReplayWeb.page is an [open-source](https://github.com/webrecorder/replayweb.page) browser-based system to replay archived web pages as accurately as possible, with interactive elements preserved.
No server necessary. Archives can be loaded as static files from anywhere a browser can connect to.
{: .fs-6 .fw-300 }
Archives can be loaded as static files from anywhere a browser can connect to, or from users local machine.
{: .fs-6 .fw-300 }
ReplayWeb.page provides the following features:
- High-Fidelity replay of web archives in [several formats](formats) and from [various locations](locations) directly in the browser.
- Several way to [explore web archives](explore): story view, page search and url search.
- On-demand, incremental [loading of large archives](loading#loading-large-files-full-vs-on-demand-loading/).
- Several options for fully-functional [offline usage](offline).
- Available as a standalone [desktop app](https://github.com/webrecorder/replayweb.page/releases) with Flash support.
- Support for [versioned embedding of web archives](embedding).