license / docs pass:

- add new docs for loading, exploring, sharing, embedding, and offline usage
- add license file / license banner

electron app:
- for now, don't bundle docs, open in native browser

embeds:
- support 'embed' option, 'replayonly' or 'full' (default if no url)

ui:
- display package.json version in info menu

package.json:
- update scripts to match README
- move electron entrypoint into extraMetadata
- add scripts for packaging as module

README: minor edits

Gemfile.lock

@@ -8,12 +8,12 @@ GEM
       eventmachine (>= 0.12.9)
       http_parser.rb (~> 0.6.0)
     eventmachine (1.2.7)
-    ffi (1.12.2)
+    ffi (1.13.0)
     forwardable-extended (2.6.0)
     http_parser.rb (0.6.0)
     i18n (0.9.5)
       concurrent-ruby (~> 1.0)
-    jekyll (3.8.6)
+    jekyll (3.8.7)
       addressable (~> 2.4)
       colorator (~> 1.0)
       em-websocket (~> 0.5)
@@ -53,7 +53,7 @@ GEM
     rb-fsevent (0.10.4)
     rb-inotify (0.10.1)
       ffi (~> 1.0)
-    rouge (3.17.0)
+    rouge (3.19.0)
     rubyzip (2.3.0)
     safe_yaml (1.0.5)
     sass (3.7.4)

LICENSE

@@ -0,0 +1,15 @@
+ReplayWeb.page
+Copyright (C) 2020  Webrecorder Software
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

README.md

@@ -23,11 +23,12 @@ The frontend is loaded from `ui.js`, while the backend service/web worker is loa
 
 This repository contains:
 - The built assets for the site hosted at https://replayweb.page/ via GitHub Pages
+- The package for npm module: https://www.npmjs.com/package/replaywebpage
 - A build system for https://replayweb.page and ReplayWeb.page App.
 - Docs hosted at: https://replayweb.page/docs
 - App releases at: https://github.com/webrecorder/replayweb.page/releases
 
-## How to Use
+## How to Use This Repo
 
 ReplayWeb.page is built as a Node package can be installed using yarn:
 
@@ -56,11 +57,15 @@ The static assets are placed in the root `index.html`, `sw.js` and `ui.js`, and
 
 For service workers to work, they must be served from either localhost or an HTTPS endpoint.
 
+See the [user docs](https://replayweb.page/docs/) for additional info about using ReplayWeb.page
+
 ## LICENSE
 
 ReplayWeb.page is made available under the AGPLv3 License.
 
-If you would like to use it under a different license, please reach out as that may be a possibility.
+[Embedding ReplayWeb.page](https://replayweb.page/docs/embedding) from published releases is encouraged.
+
+If you would like to use it under a different license or have a question, please reach out as that may be a possibility.
 
 
 ## Contributing and Bug Reports

_sass/custom/custom.scss

@@ -35,3 +35,13 @@ $nav-width-md: $nav-width;
 .rwp-blue {
   color: $rwp-blue;
 }
+
+.pad {
+  padding: 0.5rem;
+}
+
+.cap-header {
+  padding: 0 0 0 10px;
+  margin: 0;
+}
+

docs/browsing.md

@@ -1,79 +0,0 @@
----
-layout: default
-title: Browsing Web Archives
-nav_order: 1
-parent: Usage
-permalink: /docs/browsing
----
-
-# Browsing Web Archives via the ReplayWeb.page
-{: .no_toc }
-
-## Table of contents
-{: .no_toc .text-delta }
-
-1. TOC
-{:toc}
-
-
-## Loading Web Archives
-
-Using [https://replayweb.page](https://replayweb.page), you can select a web archive file from a local
-file or remote URL to load into the site.
-
-Archives loaded by ReplayWeb.page are identified by their unique URL, which can be entered directly or selected from a local file browser.
-
-
-### Remote Archives
-
-To load a remote archive, simply enter the URL of the archive and click `Replay!`.
-(See: [Supported Locations](locations.md) for details on where archives can be loaded from).
-
-The archive will be downloaded, either fully or [only as needed (if possible)](streaming-archives.md) and presented on the archive page.
-
-The system supports WARC files, as well as several other formats (See: [Supported Formats](formats.md) for more details on supported formats)
-
-Once loaded, the page is redirected to the archive collection page.
-
-
-### Local Archives
-
-Using the `Choose File` button, you can pick any local archive file to load for replay. 
-The click `Replay` to load the file. 
-
-The archive is laoded in the browser directly and *not* uploaded anywhere -- it never leaves your computer!
-
-(This process is similar to how [Webrecorder Player](https://github.com/webrecorder/webrecorder-player) works, but happens entirely in the browser.
-
-
-It will be identified by a `file://` URL and is not accessible outside your browser.
-
-
-## Archive View
-
-Once an archive is loaded, you are taken to the archive view, and you are ready to explore and replay the web archive!
-
-This view has several tabs:
-
-- **Curated Pages** - This tab exists if the archive has curated lists of pages. Currently this is available only for Webrecorder WARCs, but may be possible
-with other formats (see: [Web Archive Zip Format](webarchive-zip-format.md)
-
-- **URL Resources** - This tab allows searching the archive by URLs, as well as by MIME type. For many archives with no page or curatorial metadata, this is
-a way to explore the archive data in more detail.
-
-Once a URL is selected, you will be taken to the Replay tab which renders the archived page directly in your browser.
-
-
-## Index of Archives
-
-The ReplayWeb.page homepage lists an index of all archives loaded in your browser. 
-Once loaded, the archive remains cached in the browser for quick access. This view will be unique to your browser.
-To remove the archive from your browser, click on the 'X'. (Of course, this does not delete the archive from its original location, only your local copy of it)
-
-
-<hr>
-Next: [Sharing Links to Archived Pages](linking.md)
-
-
-
-

docs/embedding.md

@@ -0,0 +1,82 @@
+---
+layout: default
+title: Embedding ReplayWeb.page
+nav_order: 2
+permalink: /docs/embedding
+---
+
+## Embedding Web Archives with ReplayWeb.page
+
+A key goal of ReplayWeb.page is to make embedding web archives into other sites as easily as possible,
+as easy as it is to embed PDFs, for example.
+
+ReplayWeb.page provides the `<replay-web-page>` HTML tag to support embedding.
+
+Embedding requires just two parts, loading the frontend ui and the backend service worker.
+
+For example, to embed a web archive stored on s3 at `s3://webrecorder-builds/warcs/netpreserve-twitter.warc`
+(shorthand for: `https://webrecorder-builds.s3.amazonaws.com/warcs/netpreserve-twitter.warc`), you can first add
+the following snippet to your HTML page:
+
+
+{: .bg-blue-000 .text-grey-lt-000 .cap-header}
+my-web-archive-embed.html
+
+```html
+<script src="https://unpkg.com/replaywebpage@1.0.0/ui.js"></script>
+<replay-web-page source="s3://webrecorder-builds/warcs/netpreserve-twitter.warc"
+url="https://twitter.com/netpreserve"></replay-web-page>
+```
+
+This will load the frontend UI.
+
+Since ReplayWeb.page requires a service worker, it is necessary to add a service worker path
+from where the web archive will be served. Create a subdirectory (eg. `replay/`) and place the following
+one-line script
+
+
+{: .bg-blue-000 .text-grey-lt-000 .cap-header}
+./replay/sw.js
+
+```javascript
+importScripts("https://unpkg.com/replaywebpage@1.0.0/sw.js");
+```
+
+Thus, if the HTML snippet was added to `https://my-site.example.com/path/my-web-archive-embed.html`
+then the sw.js should be added such that it is at: `https://my-site.example.com/path/replay/sw.js`.
+
+That's it! Loading `https://my-site.example.com/path/my-web-archive-embed.html` should now load the web arhive.
+
+(Be sure to add sizes to the `<replay-web-page>` tag as needed to size the embed).
+
+You can replace `s3://webrecorder-builds/warcs/netpreserve-twitter.warc` with any web archive hosted on your site,
+eg.  `https://my-site.example.com/warcs/my-warc-file.warc`.
+
+{:  .fs-3 .pad .bg-grey-lt-100}
+If the file is loaded from a different origin, your site must have CORS access to download the web archive.
+
+
+### Versioning
+
+Note that the above example uses the paths as:
+
+- `https://unpkg.com/replaywebpage@1.0.0/ui.js`
+- `https://unpkg.com/replaywebpage@1.0.0/sw.js`
+
+Another alternative would be:
+
+- `https://cdn.jsdelivr.net/npm/replaywebpage@1.0.0/ui.js`
+- `https://cdn.jsdelivr.net/npm/replaywebpage@1.0.0/sw.js`
+
+These URLs point to a specific version of ReplayWeb.page software released on NPM, eg. `1.0.0`, meaning that your replay should stay stable, even if ReplayWeb.page is updated.
+
+You can choose another of ReplayWeb.page (or even try different versions) to ensure that you have tbest available replay.
+
+This addresses the potential issue of older sites breaking when web archive replay software is updated.
+
+For production use, it is advised against linking to the latest version, eg. `https://replayweb.page/ui.js`
+and `https://replayweb.page/sw.js` as these will be updated frequently and make break your embed.
+
+
+
+

docs/exploring.md

@@ -0,0 +1,66 @@
+---
+layout: default
+title: Exploring Web Archives
+nav_order: 2
+parent: Usage
+permalink: /docs/exploring
+---
+
+## Exploring Archives (Browse, Search and Replay)
+
+
+The ReplayWeb.page homepage lists an index of all loaded archives. You can also search by archive title or source
+and filter by date loaded or title.
+
+Once loaded, the archive remains cached in the browser for quick access. This view will be unique to your browser.
+To remove the archive from your browser, click on the 'X'. (Of course, this does not delete the archive from its original location, only your local copy of it)
+
+### Archive Views
+
+Although primarily designed for replay, ReplayWeb.page also offers several ways to interact with web archives.
+
+The archive view presents several tabs:
+
+- **Story** - This Story view presents lists of curated pages, as developed by the creator of the web archive. 
+This option is only shown if there is a curated story. As curated lists are not a standard part of WARC, only WARCs exported from Webrecorder.io/Conifer can have this option.
+
+The new [Web Archive Collection (WACZ)](web-archive-collection-format) can also include curated lists.
+
+- **Pages** - The Pages view presents all pages in the web archive. As pages are not a standard part of WARC format,
+generally only WARCs from Webrecorder.io/Conifer will have pages.
+
+The new [Web Archive Collection (WACZ)](web-archive-collection-format) can also store pages.
+
+
+- **Page Resources** - This view allows searching the archive by URLs, as well as by common MIME type.
+
+For many archives with no page or curatorial metadata, this is a way to explore the archive data in more detail.
+
+This view is available for all archives that only store raw data.
+
+- **Replay** - The view presents a replay of the archived web content in a mini-browser directly your browser. The view allows entering a URL directly. Clicking on links on any of the other views will switch to the **Replay** view.
+
+### Search
+
+Both the Page Resources view provide ways to search the archive directly.
+
+
+#### Searching Page Resources
+Page Resources allows searching by URL only, with additional sorting options.
+Searches can be done by exact url, by url prefix, or by any string contained in the URL.
+
+The URL Prefix option is best for searching large archives that require on-demand loading.
+The contains option will not find any URLs that have not yet been loaded.
+
+
+#### Searching Pages
+
+The Page view search includes page titles, urls and page full text search, if available.
+
+ReplayWeb.page will currently generate full ext search data from WARC pages automatically.
+
+ReplayWeb.page will soon load existing extracted full-text data as well.
+
+
+<hr>
+Next: [Sharing Links to Archived Pages](sharing.md)

docs/formats.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: 'Supported Formats'
-nav_order: 2
+nav_order: 3
 description: 'Supported Formats'
 permalink: /docs/formats
 parent: Reference
@@ -9,18 +9,20 @@ parent: Reference
 
 ## Supported Formats
 
-ReplayWeb.Page supports the following archive formats.
+ReplayWeb.Page supports the archive formats listed below.
-Format is currently determined based on the extension, though more extensive detection may be added later.
+Format is currently determined based on the file extension.
+
+The `.wacz` refers to the newly proposed [Web Archive Collection Zip Format](web-archive-collection).
 
 
 | Format  | Extensions          | Status        |     
 |:--------|:--------------------|:--------------|
 | WARC    | `.warc`, `.warc.gz` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported     |
-| HAR     | `.har`              | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported     |
+| HAR     | `.har`              | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress     |
 | WBN     | `.wbn`              | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> Experimental  | 
 | ARC     | `.arc`              | <span class="d-inline-block p-2 mr-1 v-align-middle bg-red-000"> Not Supported |
 | CDX     | `.cdx`, `.cdxj`     | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported |
-| ZIP     | `.zip`, `.waz`      | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress |
+| WACZ    | `.wacz`             | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In Progress |
 
 
 

docs/linking.md

@@ -1,8 +0,0 @@
----
-layout: default
-title: Linking to Web Archives
-nav_order: 2
-parent: Usage
-permalink: /docs/linking
----
-

docs/loading.md

@@ -0,0 +1,92 @@
+---
+layout: default
+title: Loading Web Archives
+nav_order: 1
+parent: Usage
+permalink: /docs/loading
+---
+
+# Browsing Web Archives via the ReplayWeb.page
+{: .no_toc }
+
+## Table of contents
+{: .no_toc .text-delta }
+
+1. TOC
+{:toc}
+
+
+## Loading Web Archives
+
+Using [https://replayweb.page](https://replayweb.page), you can select a web archive file from a local
+file or remote URL to load into the site.
+
+Archives loaded by ReplayWeb.page are identified by their unique URL, which can be entered directly or selected from a local file chooser.
+
+### Loading Local Web Archives
+
+Using the `Choose File...` button, you can pick any local archive file to load for replay. 
+The click `Load` to load the file. 
+
+The archive is laoded in the browser directly and *not* uploaded anywhere -- it never leaves your computer!
+
+The archive will be identified by a `file://` URL and is not accessible outside your browser. As expected,
+this URL is not shareable.
+
+{:  .fs-3 .pad .bg-grey-lt-100}
+This behavior is similar to how [Webrecorder Player](https://github.com/webrecorder/webrecorder-player) worked, but happens entirely in the browser. <br>The [ReplayWeb.page App](https://github.com/webrecorder/replayweb.page/releases) is also an option for [opening local files when offline](offline#replaywebpage-as-standalone-desktop-electron-app)
+
+
+### Loading Remote Web Archives
+
+To load a remote archive, simply enter the URL of the archive and click `Load`.
+
+{: .fs-3 .pad .bg-grey-lt-100}
+See [Supported Locations](locations) for details on where archives can be loaded from.
+
+The archive will be downloaded, either fully or [only as needed (if possible)](streaming-archives.md) and presented on the archive page.
+
+The system supports WARC files, as well as several other formats
+
+{:  .fs-3 .pad .bg-grey-lt-100}
+See: [Supported Formats](formats) for more details on supported formats.
+
+
+### Loading Remote Archives from Google Drive
+
+ReplayWeb.page also supports loading archives directly from Google Drive.
+
+ReplayWeb.page can be used to open .warc, .warc.gz files directly from google drive using the following steps.
+
+1. Install [ReplayWeb.page Google Drive Integration](https://gsuite.google.com/u/2/marketplace/app/replaywebpage/160798412227)
+
+2. Navigate to any WARC (.warc, .warc.gz) files or WACZ (Web archive collection) files that you have on google drive.
+
+3. You should see an 'Open With ReplayWeb.page' option at the top.
+
+4. Clicking on this should redirect to ReplayWeb.page and begin loading. Google Drive loaded archives will have a source URL starting with `googledrive://`.
+
+
+### Loading Large Files: Full vs On-Demand Loading
+
+Due to the nature of the WARC format, the entire file must be read on first use to generate an index. 
+
+For WARC files **>25MB**, only the index is initially stored in the browser, and the actual content is loaded 'on-demand',
+when the content is first accessed. This leads to faster loading and saves memory when dealing with large archives.
+
+[Web Archive Collection (WACZ)](web-archive-collection-format) are always loaded on-demand, as no indexing is required.
+The initial archive view should load almost instantly as a result.
+
+If an archive could not be loaded, an error will be displayed instead of the progress.
+Please report any issues encountered via [GitHub Issues](https://github.com/webrecorder/replayweb.page/issues)
+
+<hr>
+Once an archive has successfully loaded, you will be redirected to the archive view from where you can explore the archive.
+
+Next:  [Exploring the Archive](explore)
+
+
+
+
+
+

docs/locations.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: 'Supported Locations'
-nav_order: 1
+nav_order: 2
 description: 'Supported Locations'
 parent: Reference
 permalink: /docs/locations
@@ -24,9 +24,9 @@ The following is a list of currently supported URL schemes and details below:
 | `http://`        | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported   |
 | `file://`        | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported   |
 | `s3://`          | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported   |
-| `googledrive://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> In progress |
+| `googledrive://` | <span class="d-inline-block p-2 mr-1 v-align-middle bg-green-000"> Supported   |
-| `ipfs://`        | <span class="d-inline-block p-2 mr-1 v-align-middle bg-yellow-000"> Partial     |
+| `ipfs://`        | <span class="d-inline-block p-2 mr-1 v-align-middle bg-purple-000"> Planned    |
-| `dat://`         | <span class="d-inline-block p-2 mr-1 v-align-middle text-grey-lt-000 bg-purple-000"> Planned     |
+| `hyper://`       | <span class="d-inline-block p-2 mr-1 v-align-middle text-grey-lt-000 bg-purple-000"> Planned     |
 
 
 ### HTTP and HTTPS
@@ -67,11 +67,11 @@ The `ipfs://` url scheme is provide as a convenience and resolves to using the C
 A better implementation with IPFS directly is also possible.
 
 
-### DAT
+### DAT/Hyper
 {: .d-inline-block }
 
 Help Wanted
 {: .label .label-yellow }
 
-Support for `dat://` is planned to allow true peer-to-peer transfer of web archives.
+Support for `hyper://` is planned to allow true peer-to-peer transfer of web archives.
 

docs/offline.md

@@ -0,0 +1,41 @@
+---
+layout: default
+title: Offline Use
+nav_order: 4
+parent: Usage
+permalink: /docs/offline
+---
+
+# Offline Use
+
+ReplayWeb.page is available for offline use in several ways.
+
+## ReplayWeb.page Offline Cache
+
+After ReplayWeb.page is loaded, it can automatically be used offline without requiring an internet connection.
+
+If a connection is not available, all web archive already loaded will continue to be available.
+
+[Loading local archives](loading#loading-local-web-archives) should fully work without an internet connection.
+
+## ReplayWeb.page as Progress Web App
+
+ReplayWeb.page can also be installed as [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) and be available as a standalone app on desktop and mobile.
+
+This is still experimental, but should provide the same behavior as regular https://replayweb.page/
+
+## ReplayWeb.page as Standalone Desktop (Electron) App
+
+ReplayWeb.page is also available as a standalone Electron App. In this mode, ReplayWeb.page
+ships its own version of Chrome.
+
+Although the Web Page/Progressive Web App versions have almost the same features, the standalone App is useful for two primary reasons:
+
+1) For viewing web archives that contain Flash. Flash is not supported by default in most browsers and will soon be removed. The ReplayWeb.page App will bundle a Flash plugin to continue to make Flash accessible for web archives.
+
+2) Primary use for viewing offline web archive. If your primary use is to view web archives stored locally,
+the ReplayWeb.page App is a better option, as it provides direct access to the file system.
+Once loaded, ReplayWeb.page App can continue to load web archives from disk.
+The desktop app fully replaces [Webrecorder Player](https://github.com/webrecorder/webrecorder-player)
+
+When using [https://replayweb.page](https://replayweb.page), you may be asked to re-open a local file because a web site can not have persistent access to your file system.

docs/reference.md

@@ -1,11 +1,11 @@
 ---
 layout: default
 title: Reference
-nav_order: 2
+nav_order: 3
 has_children: true
 permalink: /docs/reference
 ---
 
 ## Reference
 
-A guide to the file formats, source locations and query arguments supported by ReplayWeb.Page
+A guide to the file formats, specifications, source locations and other APIs to be supported by ReplayWeb.Page

docs/sharing.md

@@ -0,0 +1,74 @@
+---
+layout: default
+title: Sharing Web Archives
+nav_order: 3
+parent: Usage
+permalink: /docs/sharing
+---
+
+## Sharing Links
+
+ReplayWeb.page links are designed to be shareable with others, as long as they have access.
+
+You can share links to any view, including Page or Page Resoures search, or Replay view.
+
+ReplayWeb.page encodes the web archive source and view parameters in the URL query and hash string.
+
+Simply copy the entire location url, and you can share with others!
+
+ReplayWeb.page will parse the URL, load the source web archive, then load the view or replay from that archive.
+
+For example, given the following URL:
+
+```
+https://replayweb.page/?source=s3://webrecorder-builds/warcs/netpreserve-twitter.warc#view=replay&url=https%3A%2F%2Ftwitter.com%2Fnetpreserve&ts=20190603053135
+```
+
+ReplayWeb.page will load the web archive from `s3://webrecorder-builds/warcs/netpreserve-twitter.warc`, switch to 
+the replay view via `https://twitter.com/netpreserve` at timestamp `20190603053135`
+
+The following URL will load the same web archive, and show the search results for Pages that contain the word 'WARC':
+
+```
+https://replayweb.page/?source=s3://webrecorder-builds/warcs/netpreserve-twitter.warc#view=pages&query=WARC
+```
+
+### No Local Files
+
+For obvious reasons, URLs that have a `source=file://...` can not be shared as they refer to a local file on your machine.
+Loading that URL by anyone else will result in an error.
+
+{:  .fs-3 .pad .bg-grey-lt-100}
+A future goal is to provide an option to share local web archives peer-to-peer via the DAT `hyper://` protocol.
+
+
+### Sharing Google Drive Archives
+
+An archive loaded from google drive can be shared with others, as long as they have access.
+
+Google Drive Archive will have a `source=googledrive://...`.
+
+Due to restrictions of Google Drive, you will likely be required to authorize with your Google Account to access an archive from Google Drive.
+
+Public files shared from Google Drive can sometimes be loaded without requiring authorization, but this is only supported
+for small files.
+
+To share a Google Drive web archive with colleagues and friends:
+
+1. Ensure they you have [loaded the web archive from Google Drive](loading#loading-remote-archives-from-google-drive)
+
+2. Ensure that your friends/colleagues also have the [ReplayWeb.page Google Drive Integration](https://gsuite.google.com/u/2/marketplace/app/replaywebpage/160798412227) installed.
+
+3. Share the file with them via Google Drive. This is the most surefire way to work.
+
+OR.
+
+3. You can share the direct ReplayWeb.page URL directly. This does not always work, and you may need to share the Google Drive link instead.
+
+
+In this way, web archives can be shared within private networks using existing access control settings in Google Drive.
+
+<hr>
+Next: Offline Usage Options
+
+

docs/usage.md

@@ -10,11 +10,26 @@ permalink: /docs/index.html
 # ReplayWeb.Page
 ## Serverless / Client-Side Web Archive Replay
 
-A browser-based system to replay archived web pages as accurately as possible, with all interactive elements preserved.
 {: .fs-6 .fw-300 }
+ReplayWeb.page is an [open-source](https://github.com/webrecorder/replayweb.page) browser-based system to replay archived web pages as accurately as possible, with interactive elements preserved.
 
-No server necessary. Archives can be loaded as static files from anywhere a browser can connect to.
 {: .fs-6 .fw-300 }
+Archives can be loaded as static files from anywhere a browser can connect to, or from users local machine.
+
+{: .fs-6 .fw-300 }
+ReplayWeb.page provides the following features:
+
+- High-Fidelity replay of web archives in [several formats](formats) and from [various locations](locations) directly in the browser.
+
+- Several way to [explore web archives](explore): story view, page search and url search.
+
+- On-demand, incremental [loading of large archives](loading#loading-large-files-full-vs-on-demand-loading/).
+
+- Several options for fully-functional [offline usage](offline).
+
+- Available as a standalone [desktop app](https://github.com/webrecorder/replayweb.page/releases) with Flash support.
+
+- Support for [versioned embedding of web archives](embedding).
 
 
 

package.json

@@ -3,7 +3,6 @@
   "productName": "ReplayWeb.page",
   "version": "1.0.0",
   "description": "Serverless Web Archive Replay",
-  "main": "dist/electron.js",
   "repository": "https://github.com/webrecorder/replayweb.page",
   "homepage": "https://replayweb.page/",
   "author": "Ilya Kreymer <ikreymer@gmail.com>",
@@ -25,6 +24,7 @@
     "electron": "9.0.3",
     "electron-builder": "^22.7.0",
     "electron-notarize": "^1.0.0",
+    "http-server": "^0.12.3",
     "mini-css-extract-plugin": "^0.9.0",
     "node-sass": "^4.14.0",
     "raw-loader": "^4.0.1",
@@ -36,15 +36,21 @@
     "webpack-cli": "^3.3.11",
     "webpack-dev-server": "^3.10.3"
   },
+  "files": [
+    "/src",
+    "sw.js",
+    "ui.js"
+  ],
   "scripts": {
     "build": "webpack --mode production",
     "build-dev": "webpack --mode development",
+    "build-docs": "bundle install; bundle exec jekyll build",
     "start-dev": "webpack-dev-server --mode development",
+    "start-prod": "http-server -p 9990",
     "pack": "CSC_IDENTITY_AUTO_DISCOVERY=false electron-builder --publish never",
     "pack-signed": "electron-builder",
     "start-electron": "NODE_ENV=development electron ./dist/electron.js",
-    "build-electron": "yarn run build && yarn run docs && yarn run pack",
+    "start-jekyll": "yarn run build-docs && bundle exec jekyll serve",
-    "docs": "bundle install; bundle exec jekyll build",
     "dist": "yarn run build && yarn run pack",
     "release": "yarn run build && electron-builder"
   },
@@ -54,15 +60,14 @@
     "asar": true,
     "appId": "net.webrecorder.replaywebpage",
     "artifactName": "${productName}-${version}.${ext}",
+    "extraMetadata": {
+      "main": "dist/electron.js"
+    },
     "files": [
       "!**/node_modules/**/*",
       "dist/*.js",
       "*.js",
       "*.html",
-      {
-        "from": "_site/docs",
-        "to": "docs"
-      },
       {
         "from": "_site/assets",
         "to": "assets"

src/appmain.js

@@ -162,6 +162,10 @@ class App extends LitElement
             <div class="navbar-item">
               Collection Id:&nbsp;<b>${this.collInfo.coll}</b>
             </div>
+            <hr class="navbar-divider">
+            <div class="navbar-item">
+              ReplayWeb.page Version: ${__VERSION__}
+            </div>
           </div>
         </div>
         ` : ``}

src/electron-main.js

@@ -3,7 +3,7 @@
 import fetch from 'node-fetch';
 import { Headers } from 'node-fetch';
 
-import {app, session, BrowserWindow, ipcMain} from 'electron';
+import {app, session, BrowserWindow, ipcMain, shell} from 'electron';
 
 import path from 'path';
 import fs from 'fs';
@@ -20,9 +20,6 @@ const STATIC_PREFIX = "http://localhost:5471/";
 
 const REPLAY_PREFIX = STATIC_PREFIX + "wabac/";
 
-const HTML_TYPE = 'text/html; charset="utf-8"';
-const JS_TYPE = 'application/javascript; charset="utf-8"';
-
 const URL_RX = /([^\/]+)\/([\d]+)(?:\w\w_)?\/(.*)$/;
 
 const appPath = app.getAppPath();
@@ -290,7 +287,8 @@ function createWindow () {
       plugins: true,
       preload: path.join(__dirname, 'preload.js'),
       nativeWindowOpen: true,
-      contextIsolation: true
+      contextIsolation: true,
+      enableRemoteModule: false
     }
   }).once('ready-to-show', () => {
 
@@ -315,6 +313,17 @@ function createWindow () {
   }
 }
 
+app.on('web-contents-created', (event, contents) => {
+  contents.on('new-window', async (event, navigationUrl) => {
+    
+    // load docs in native browser for now
+    if (navigationUrl === STATIC_PREFIX + "docs") {
+      event.preventDefault();
+      await shell.openExternal("https://replayweb.page/docs/");
+    }
+  });
+});
+
 // This method will be called when Electron has finished
 // initialization and is ready to create browser windows.
 // Some APIs can only be used after this event occurs.

src/embed.js

@@ -21,6 +21,7 @@ class Embed extends LitElement
     this.paramString = null;
     this.deepLink = false;
     this.swInited = false;
+    this.embed = null;
 
     this.doRegister();
   }
@@ -30,6 +31,8 @@ class Embed extends LitElement
       url: { type: String },
       ts: { type: String },
       source: { type: String },
+      view: { type: String },
+      embed: { type: String },
 
       replayBase: { type: String },
       swName: { type: String },
@@ -101,11 +104,15 @@ class Embed extends LitElement
         changedProperties.has("ts") ||
         changedProperties.has("source")) {
 
+      if (this.embed === null) {
+        this.embed = this.url ? "replayonly" : "";
+      }
+
       this.paramString = new URLSearchParams({
         source: this.source,
         customColl: this.coll,
         config: this.config,
-        embed: "true"
+        embed: this.embed,
       }).toString();
   
       this.hashString = new URLSearchParams({

webpack.config.js

@@ -91,8 +91,10 @@ const browserConfig = (env, argv) => {
         __SW_NAME__: JSON.stringify('sw.js'),
         __APP_FILE_SERVE_PREFIX__ : JSON.stringify(APP_FILE_SERVE_PREFIX),
         __HELPER_PROXY__ : JSON.stringify(HELPER_PROXY),
-        __GDRIVE_CLIENT_ID__ : JSON.stringify(GDRIVE_CLIENT_ID)
+        __GDRIVE_CLIENT_ID__ : JSON.stringify(GDRIVE_CLIENT_ID),
+        __VERSION__: JSON.stringify(require("./package.json").version)
       }),
+      new webpack.BannerPlugin('[name].js is part of ReplayWeb.page (https://replayweb.page) Copyright (C) 2020, Webrecorder Software. Licensed under the Affero General Public License v3.')
     ],
 
     module: {
@@ -114,7 +116,7 @@ const browserConfig = (env, argv) => {
         loaders: ['raw-loader'],
       }
       ]
-    }
+    },
   }
 };
 

yarn.lock

@@ -630,6 +630,11 @@ base@^0.11.1:
     mixin-deep "^1.2.0"
     pascalcase "^0.1.1"
 
+basic-auth@^1.0.3:
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/basic-auth/-/basic-auth-1.1.0.tgz#45221ee429f7ee1e5035be3f51533f1cdfd29884"
+  integrity sha1-RSIe5Cn37h5QNb4/UVM/HN/SmIQ=
+
 batch@0.6.1:
   version "0.6.1"
   resolved "https://registry.yarnpkg.com/batch/-/batch-0.6.1.tgz#dc34314f4e679318093fc760272525f94bf25c16"
@@ -1180,6 +1185,11 @@ color-name@~1.1.4:
   resolved "https://registry.yarnpkg.com/color-name/-/color-name-1.1.4.tgz#c2a09a87acbde69543de6f63fa3995c826c536a2"
   integrity sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==
 
+colors@^1.4.0:
+  version "1.4.0"
+  resolved "https://registry.yarnpkg.com/colors/-/colors-1.4.0.tgz#c50491479d4c1bdaed2c9ced32cf7c7dc2360f78"
+  integrity sha512-a+UqTh4kgZg/SlGvfbzDHpgRu7AAQOmmqRHJnxhRZICKFUT91brVhNNt58CMWU9PsBbv3PDCZUHbVxuDiH2mtA==
+
 combined-stream@^1.0.6, combined-stream@~1.0.6:
   version "1.0.8"
   resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.8.tgz#c3d45a8b34fd730631a110a8a2520682b31d5a7f"
@@ -1326,6 +1336,11 @@ core-util-is@1.0.2, core-util-is@~1.0.0:
   resolved "https://registry.yarnpkg.com/core-util-is/-/core-util-is-1.0.2.tgz#b5fd54220aa2bc5ab57aab7140c940754503c1a7"
   integrity sha1-tf1UIgqivFq1eqtxQMlAdUUDwac=
 
+corser@^2.0.1:
+  version "2.0.1"
+  resolved "https://registry.yarnpkg.com/corser/-/corser-2.0.1.tgz#8eda252ecaab5840dcd975ceb90d9370c819ff87"
+  integrity sha1-jtolLsqrWEDc2XXOuQ2TcMgZ/4c=
+
 create-ecdh@^4.0.0:
   version "4.0.3"
   resolved "https://registry.yarnpkg.com/create-ecdh/-/create-ecdh-4.0.3.tgz#c9111b6f33045c4697f144787f9254cdc77c45ff"
@@ -1692,6 +1707,16 @@ ecc-jsbn@~0.1.1:
     jsbn "~0.1.0"
     safer-buffer "^2.1.0"
 
+ecstatic@^3.3.2:
+  version "3.3.2"
+  resolved "https://registry.yarnpkg.com/ecstatic/-/ecstatic-3.3.2.tgz#6d1dd49814d00594682c652adb66076a69d46c48"
+  integrity sha512-fLf9l1hnwrHI2xn9mEDT7KIi22UDqA2jaCwyCbSUJh9a1V+LEUSL/JO/6TIz/QyuBURWUHrFL5Kg2TtO1bkkog==
+  dependencies:
+    he "^1.1.1"
+    mime "^1.6.0"
+    minimist "^1.1.0"
+    url-join "^2.0.5"
+
 ee-first@1.1.1:
   version "1.1.1"
   resolved "https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz#590c61156b0ae2f4f0255732a158b266bc56b21d"
@@ -2622,6 +2647,11 @@ hash.js@^1.0.0, hash.js@^1.0.3:
     inherits "^2.0.3"
     minimalistic-assert "^1.0.1"
 
+he@^1.1.1:
+  version "1.2.0"
+  resolved "https://registry.yarnpkg.com/he/-/he-1.2.0.tgz#84ae65fa7eafb165fddb61566ae14baf05664f0f"
+  integrity sha512-F/1DnUGPopORZi0ni+CvrCgHQ5FyEAHRLSApuYWMmrbSwoN2Mn/7k+Gl38gJnR7yyDZk6WLXwiGod1JOWNDKGw==
+
 hi-base32@^0.5.0:
   version "0.5.0"
   resolved "https://registry.yarnpkg.com/hi-base32/-/hi-base32-0.5.0.tgz#61329f76a31f31008533f1c36f2473e259d64571"
@@ -2736,6 +2766,31 @@ http-proxy@^1.17.0:
     follow-redirects "^1.0.0"
     requires-port "^1.0.0"
 
+http-proxy@^1.18.0:
+  version "1.18.1"
+  resolved "https://registry.yarnpkg.com/http-proxy/-/http-proxy-1.18.1.tgz#401541f0534884bbf95260334e72f88ee3976549"
+  integrity sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==
+  dependencies:
+    eventemitter3 "^4.0.0"
+    follow-redirects "^1.0.0"
+    requires-port "^1.0.0"
+
+http-server@^0.12.3:
+  version "0.12.3"
+  resolved "https://registry.yarnpkg.com/http-server/-/http-server-0.12.3.tgz#ba0471d0ecc425886616cb35c4faf279140a0d37"
+  integrity sha512-be0dKG6pni92bRjq0kvExtj/NrrAd28/8fCXkaI/4piTwQMSDSLMhWyW0NI1V+DBI3aa1HMlQu46/HjVLfmugA==
+  dependencies:
+    basic-auth "^1.0.3"
+    colors "^1.4.0"
+    corser "^2.0.1"
+    ecstatic "^3.3.2"
+    http-proxy "^1.18.0"
+    minimist "^1.2.5"
+    opener "^1.5.1"
+    portfinder "^1.0.25"
+    secure-compare "3.0.1"
+    union "~0.5.0"
+
 http-signature@~1.2.0:
   version "1.2.0"
   resolved "https://registry.yarnpkg.com/http-signature/-/http-signature-1.2.0.tgz#9aecd925114772f3d95b65a60abb8f7c18fbace1"
@@ -3648,7 +3703,7 @@ mime-types@^2.1.12, mime-types@^2.1.27, mime-types@~2.1.17, mime-types@~2.1.19,
   dependencies:
     mime-db "1.44.0"
 
-mime@1.6.0:
+mime@1.6.0, mime@^1.6.0:
   version "1.6.0"
   resolved "https://registry.yarnpkg.com/mime/-/mime-1.6.0.tgz#32cd9e5c64553bd58d19a568af452acff04981b1"
   integrity sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==
@@ -3695,7 +3750,7 @@ minimatch@^3.0.4, minimatch@~3.0.2:
   dependencies:
     brace-expansion "^1.1.7"
 
-minimist@^1.1.3, minimist@^1.2.0, minimist@^1.2.5:
+minimist@^1.1.0, minimist@^1.1.3, minimist@^1.2.0, minimist@^1.2.5:
   version "1.2.5"
   resolved "https://registry.yarnpkg.com/minimist/-/minimist-1.2.5.tgz#67d66014b66a6a8aaa0c083c5fd58df4e4e97602"
   integrity sha512-FM9nNUYrRBAELZQT3xeZQ7fmMOBg6nWNmJKTcgsJeaLstP/UODVpGsr5OhXhhXg6f+qtJ8uiZ+PUxkDWcgIXLw==
@@ -4052,6 +4107,11 @@ once@^1.3.0, once@^1.3.1, once@^1.4.0:
   dependencies:
     wrappy "1"
 
+opener@^1.5.1:
+  version "1.5.1"
+  resolved "https://registry.yarnpkg.com/opener/-/opener-1.5.1.tgz#6d2f0e77f1a0af0032aca716c2c1fbb8e7e8abed"
+  integrity sha512-goYSy5c2UXE4Ra1xixabeVh1guIX/ZV/YokJksb6q2lubWu6UbvPQ20p542/sFIll1nl8JnCyK9oBaOcCWXwvA==
+
 opn@^5.5.0:
   version "5.5.0"
   resolved "https://registry.yarnpkg.com/opn/-/opn-5.5.0.tgz#fc7164fab56d235904c51c3b27da6758ca3b9bfc"
@@ -4574,6 +4634,11 @@ qs@6.7.0:
   resolved "https://registry.yarnpkg.com/qs/-/qs-6.7.0.tgz#41dc1a015e3d581f1621776be31afb2876a9b1bc"
   integrity sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==
 
+qs@^6.4.0:
+  version "6.9.4"
+  resolved "https://registry.yarnpkg.com/qs/-/qs-6.9.4.tgz#9090b290d1f91728d3c22e54843ca44aea5ab687"
+  integrity sha512-A1kFqHekCTM7cz0udomYUoYNWjBebHm/5wzU/XqrBRBNWectVH0QIiN+NEcZ0Dte5hvzHwbr8+XQmguPhJ6WdQ==
+
 qs@~6.5.2:
   version "6.5.2"
   resolved "https://registry.yarnpkg.com/qs/-/qs-6.5.2.tgz#cb3ae806e8740444584ef154ce8ee98d403f3e36"
@@ -4988,6 +5053,11 @@ search-params@3.0.0:
   resolved "https://registry.yarnpkg.com/search-params/-/search-params-3.0.0.tgz#dbc7c243058e5a33ae1e9870be91f5aced4100d8"
   integrity sha512-8CYNl/bjkEhXWbDTU/K7c2jQtrnqEffIPyOLMqygW/7/b+ym8UtQumcAZjOfMLjZKR6AxK5tOr9fChbQZCzPqg==
 
+secure-compare@3.0.1:
+  version "3.0.1"
+  resolved "https://registry.yarnpkg.com/secure-compare/-/secure-compare-3.0.1.tgz#f1a0329b308b221fae37b9974f3d578d0ca999e3"
+  integrity sha1-8aAymzCLIh+uN7mXTz1XjQypmeM=
+
 select-hose@^2.0.0:
   version "2.0.0"
   resolved "https://registry.yarnpkg.com/select-hose/-/select-hose-2.0.0.tgz#625d8658f865af43ec962bfc376a37359a4994ca"
@@ -5820,6 +5890,13 @@ union-value@^1.0.0:
     is-extendable "^0.1.1"
     set-value "^2.0.1"
 
+union@~0.5.0:
+  version "0.5.0"
+  resolved "https://registry.yarnpkg.com/union/-/union-0.5.0.tgz#b2c11be84f60538537b846edb9ba266ba0090075"
+  integrity sha512-N6uOhuW6zO95P3Mel2I2zMsbsanvvtgn6jVqJv4vbVcz/JN0OkL9suomjQGmWtxJQXOCqUJvquc1sMeNz/IwlA==
+  dependencies:
+    qs "^6.4.0"
+
 uniq@^1.0.1:
   version "1.0.1"
   resolved "https://registry.yarnpkg.com/uniq/-/uniq-1.0.1.tgz#b31c5ae8254844a3a8281541ce2b04b865a734ff"
@@ -5905,6 +5982,11 @@ urix@^0.1.0:
   resolved "https://registry.yarnpkg.com/urix/-/urix-0.1.0.tgz#da937f7a62e21fec1fd18d49b35c2935067a6c72"
   integrity sha1-2pN/emLiH+wf0Y1Js1wpNQZ6bHI=
 
+url-join@^2.0.5:
+  version "2.0.5"
+  resolved "https://registry.yarnpkg.com/url-join/-/url-join-2.0.5.tgz#5af22f18c052a000a48d7b82c5e9c2e2feeda728"
+  integrity sha1-WvIvGMBSoACkjXuCxenC4v7tpyg=
+
 url-parse-lax@^3.0.0:
   version "3.0.0"
   resolved "https://registry.yarnpkg.com/url-parse-lax/-/url-parse-lax-3.0.0.tgz#16b5cafc07dbe3676c1b1999177823d6503acb0c"