Liveweb Proxy

Liveweb proxy is a component of Internet Archive’s wayback machine project.

The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.

Note

The liveweb project is under active development, so this documentation may not be up-to-date.

Installation

Liveweb proxy can be installed using pip:

$ pip install liveweb

or, with easy_install

$ easy_install liveweb

See Development Setup if you want to work with source code.

Running liveweb-proxy

Liveweb proxy can be run using:

$ liveweb-proxy

To start liveweb-proxy on a different port:

$ liveweb-proxy -p 8080

To load settings from a config file:

$ liveweb-proxy -c liveweb.ini

To see the available command-line options:

$ liveweb-proxy --help

See Configuration section for the available config settings and command line options.

Advanced Usage

Under the hood, liveweb proxy used uwsgi as the http server.

If you want to tweak uwsgi parameters, you can start liveweb as:

$ uwsgi --master --single-interpreter --lazy --wsgi liveweb.main --processes 1 --threads 10 --http localhost:7070

The values of --processes, --threads and --http options can be changed as needed and more options can be added too.

You may have to specify -H virtualenv_home if you using a virtualenv.

Project Versions

Table Of Contents

Next topic

Development Setup

This Page