ICM Workshop on Web Caching

Martin Hamilton, Loughborough University

This session stems from a discussion on HTTP client resilience on the www-proxy mailing list.

Many thanks to Peter Lister, Neil Smith, Stewart Brodie and Jon Knight for their comments.

Propositions...

Proxy HTTP clients, e.g. in WWW browsers and proxy servers which have been chained to other proxy servers, should allow for:

direct connections to be used in the event of the proxy or proxies failing
multiple proxy servers to be used (in whatever configuration), so that clients aren't wholly dependent on a single proxy
proxying to be disabled for selected requests, e.g. based on the domain name component of the request if present
arbitrary protocol schemes to be proxied, i.e. including ones which the software does not have built-in knowledge of

and?

Feedback...

From some discussion between Peter Lister and Neil Smith...

It should be possible to configure logs and feedback to the user to display any combination of details, including at least (a) which URLs are being requested, and (b) which servers were used to satisfy the requests and whether the request directly or by proxy.

Neil Smith adds... (on the subject of selecting from multiple proxies)

Clients need to be able to handle both cases, where there are a number of tightly coupled machines (a cluster) acting as a single proxy, and a number of alternative proxies.

Issues...

Some resolvers only return a single IP address, even if the server has several.
Some clients only try a single IP address, even if the resolver returns several
DNS entries are cached, but TTLs aren't honoured
nscd :-(
How should client code deal with multiple IP addresses anyway? e.g. rotate to a new address on each successive connection, or just on failure ?
Weed out bad addresses for good, or just temporarily ?

Conclusions...

Looking for comments on the above!
Any further points ?
Role of SRV ?
(draft-gulbrandsen-dns-rr-srvcs-03.txt)
Proposed HTTP IMPLEMENTORS GUIDE from IETF's HTTP group would be a good place to record concerns over resilience ?