Liveweb proxy is a component of Internet Archive’s wayback machine project.
The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Note
The liveweb project is under active development, so this documentation may not be up-to-date.
Liveweb proxy can be installed using pip:
$ pip install liveweb
or, with easy_install
$ easy_install liveweb
See Development Setup if you want to work with source code.
Liveweb proxy can be run using:
$ liveweb-proxy
To start liveweb-proxy on a different port:
$ liveweb-proxy -p 8080
To load settings from a config file:
$ liveweb-proxy -c liveweb.ini
To see the available command-line options:
$ liveweb-proxy --help
See Configuration section for the available config settings and command line options.
Under the hood, liveweb proxy used uwsgi as the http server.
If you want to tweak uwsgi parameters, you can start liveweb as:
$ uwsgi --master --single-interpreter --lazy --wsgi liveweb.main --processes 1 --threads 10 --http localhost:7070
The values of --processes, --threads and --http options can be changed as needed and more options can be added too.
You may have to specify -H virtualenv_home if you using a virtualenv.