The World Free Web
Compare FreeNet to the current WWW architecture. As information on a Web server becomes more popular, it becomes more difficult for users to access. Witness the impressive "Slashdot effect" which occurs when thousands of users suddenly overwhelm a Web server. The Slashdot effect is caused by the centralization of information. When information is centrally located, this location becomes a single point of failure for the distribution of that information. The irony of the Slashdot effect is that it is caused by users making thousands of copies of the relevant information. It's not as if the information has become scarce -- quite the opposite. Users ought to be able to share that information with each other, decentralizing it. [Editor's note: In fact, this already happens. What do you see almost immediately in the comments to a story that links to a movie trailer or the photos of Hemos's wedding? "Here's a mirror" and "Here's another". Making this more efficient by turning it into part of the system would actually just be the next step in an already-established practice.]
I propose that FreeNet be integrated into the Mozilla cache structure, allowing users to form a sort of "browsers' cooperative" in which pages are freely shared in a giant collective caching structure. (Of course, this should not be limited just to the Mozilla browser, but I think it's a good starting point.)
I have given the structure of this cooperative some thought, and I want to give an overview of how I think the system should work. First comes a technical overview, which details the relatively simple integration work that needs to be done. Second, I will give a more abstract view of how I think the social structure of such a cooperative ought to be formed. I have dubbed this system the World Free Web (or WFW, not to be confused with the WWF or WCW).
Technical Overview
When a user makes a request for a page using this new enhanced WFW browser, three things would happen simultaneously:
- The browser makes an HTTP request to the requisite server.
- The browser checks the on-disk and in-memory caches.
- The browser submits a FreeNet request to the local FreeNet node.
Now, whichever of these methods returns a valid result first is displayed to the user. The user need not have any knowledge of which method was used, although if they get outdated or garbage data, they can always hit "shift reload" which should force the browser to use method #1 to re-fetch the data. This is similar to the way many proxy servers work today. Behind the scenes, as each page is inserted into the browser cache, it is also inserted into the local FreeNet node. The entire process is transparent to the end user.
In the course of normal operation, this whole scheme will behave in a very similar way to normal Web browsing. There are really only two cases in which the FreeNet node would provide a page faster than the HTTP request:
- When the site in question is down or under heavy load.
- When there is high network lag between the user and the site.
Philosophically, this scheme has one primary benefit: Each user of the Web, even a non-techie type, becomes a contributor to the network infrastructure instead of simply a drain on resources. This is much like the old days of Usenet, when each person shared her news feed with others. More on this below.
Practically, there are many more extended benefits. For instance, one of the problems with distributed systems such as FreeNet is the lack of feedback or ratings on the quality of the information. The WFW can automatically provide reliable feedback on the validity of information. If a user hits the "super reload" button after getting a page from FreeNet, this page is likely to be of suspect quality (it is either a bogus result or out-of-date). Large Web sites will no longer have a monopoly on the ability to handle a large number of users. This is just an example of the kinds of things users can do when they band together.
But it gets better. Once this starts to catch on, proxy servers and Web servers can be adapted to start participating in the system. Imagine a new HTTP response code that indicates that the server is too busy to handle your request right now, but that the data you want was just inserted into FreeNet with a given key. Small sites get to leverage the bandwidth and storage of their users to reduce costs.
FreeNet itself benefits in a number of ways. Since many more people are using FreeNet just by using their browsers, the amount of information and overall storage capacity in FreeNet is increased by several orders of magnitude. All the virtues of FreeNet's design become stronger as the number of users increases. Having more nodes increases the overall resilience of the network to attack, and having nodes run transparently by "normal" users makes it harder to accuse FreeNet users of engaging in suspicious activity.
Social Organization
In order to be successful, I believe the WFW should work as a true grassroots user movement. In order to support this, the WFW will have to add slightly to the underlying FreeNet protocol. However, a few things should be noted. First, WFW nodes could still act as normal FreeNet nodes, performing all the operations that the typical FreeNet node would. The rules I am about to outline would only apply to WFW nodes talking to other WFW nodes, and need not apply when they are talking to normal FreeNet nodes.
First of all, one of the big problems with FreeNet as it currently stands is the bootstrapping process of finding out about other FreeNet nodes. Currently, FreeNet maintains an optional central repository of nodes which is available via the Web. This is not a great long-term solution, as it reintroduces centralization into a system that should be fully distributed. My proposal is that the WFW be a closed "club" structure. In order to join, you have to get an existing member to sponsor you. In many cases, this member could just be your ISP, but it does not need to be.
Each WFW node could have an ACL that keeps track of other nodes that the current node is willing to accept requests from. When a node is introduced into the system via a sponsorship, at first this node will only be allowed to make requests via the sponsoring node. The node will also handle requests through its parent node, but as it fulfills these requests (and hence, becomes more and more useful to the rest of the network), other nodes will start to accept direct connections from it. This produces the proper incentives to marginalize the effects of spammers. If you are going to sponsor people, your node will be the primary victim of any malicious activity they engage in, and you will be able to cut off their access if they do engage in such behavior. Only after a node has proved its utility to the rest of the network will it gradually be brought closer to the strongly-connected center, and if it starts to change its behavior, it will gradually be pushed out towards the periphery.
Clearly, there is much more work to be done. The WFW is a first step towards accomplishing a more intelligent mainstream net architecture which recognizes that information cannot and should not be controlled by an elite few. But this is just an outline, a sketch of what's coming. I am hoping to get people involved in a development effort -- a few from the FreeNet team, a few from the Mozilla team at first -- but then there's plenty more work to be done. If this is to succeed, it will have to be a community effort. Consider this your official invitation. If you'd like to get involved, the project has a home at http://enzyme.sourceforge.net/WFW/.
While you're thinking about these issues, you might want to check out Professor David Gelernter's latest manifesto, The Second Coming.
Eric Ries (eries@CatalystRecruiting.com) is working on a BS in Computer Science and a BA in Philosophy at Yale University. He is currently CTO of the Internet startup company Catalyst Recruiting (http://www.CatalystRecruiting.com/) and its cousin, the Enzyme open-source project (http://enzyme.sourceforge.net/). His previous work experience ranges from Microsoft to the San Diego Supercomputer Center. He has been published on Java and other topics in both books and magazines. He was co-author of The Black Art of Java Game Programming, among others, and was the Games & Graphics editor for the Java Developer's Journal. His complete resume is available at http://i.am/EricRies/.
T-Shirts and Fame!
We're eager to find people interested in writing editorials on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an editorial gets a freshmeat t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.
rediscovering distributed caching
Or we could just install Squid proxies everywhere and get all the benefits of a hierarchical proxy network without having to buy into FreeNet's political agenda and other problems. What happens when the content gets updated?
Why not do it in the proxy ?
Why not just implement it in a proxy (such as Squid) instead of increasing the complexity of the browser ?
It also seems that this kind of functionallity seems better fit to be in a proxy instead of the browser ?
And a note to the commenter about "buy into the policatal agenda":
Remember that by using and/or developing GPL software you are "buying into the political agenda" of Free Software Foundation :)
squid
I've used squid at home and at work. Where I've had a static IP and
reasonable bandwidth, I've joined the cache hierarchy. If this were going to be a widely-used solution, it would have done so already. It hasn't because:
It requires an administrator to install squid and maintain it.
It requires the users to configure their browsers to use the proxy (and to know how to stop using it if it goes down).
It doesn't scale well, either; it relies on the CPU, memory, and bandwidth of the proxy server instead of distributing the load among the clients.
Eric's suggestion could work because it doesn't have any of these limitations. It doesn't create any burden of work for the user or the admin. People just install the browser and surf like they always have. Just by surfing, they contribute to making everything faster. They don't have to know anything about computers; they'll just know that suddenly they can get to sites that were unreachable before. We benefit from all the AOL users who would never, ever figure out how to configure the proxy settings in their browsers.
And it scales better. If squid were widely used, people would be overloading their local cache with requests instead of overloading the origin site. This would help, but not nearly as much as if everyone got a copy from the next-door neighbor instead of the city cache. If everyone's helping to serve information as well as suck it, pretty soon, you've got distributed.net-style surfing.
What's wrong with this?
I can see some potential problems here....
You gave the example of Usenet feeds. However, Usenet messages are static once they are sent - you can't "back up" in the news feed stream and edit a message in any way. Web pages, on the other hand, can be changed at any time. I could write a script to do a tweak to a (theoretical) gig of documents on my server, then your service would have to re-propagate these throughout the network. Keeping up with the web would be a huge job.
What you're talking about sounds like, basically, not only indexing and caching the entire web, or at least the portion running your software, a job that huge companies with enormous server farms and massive budgets have not even come close to accomplishing.
And, what about online scripts? Host sites would have to be able to generate dynamic content, a rather large part of the web today (tangent: any stats on how much of the web actually is dynamic? that could be interesting). This is really not very feasible, unless you restrict users to 1 language and 1 feature set. Most sites use a pretty wide variety of languages / Apache modules / add-ons / shiny super duper new scripting gadgets. And if you do allow unrestricted scripting, that brings up security concerns.
The benefits of this would be marginal; most sites don't max out their bandwidth / hardware capacities very often, and if they do, it's time for an upgrade. So this would only help for huge peaking traffic - say, the /. effect.
Instead of taking on this huge project, which you cite as a way to combat the /. effect and has limited other usefulness, why don't you just whip up some scripts to go out and mirror all the static content from every page that Slashdot links to? *grin*
And if you do try to pull this off...best of luck, it could be a neat project after some tweaking in the concept.
problems with the model...?
Jeff has pretty adequately covered the advantages of a scheme like this over any kind of distributed proxy/server combination - namely that overactive AOL "me-too"ers can contribute to the network.
"then your service would have to re-propagate these throughout the network. Keeping up with the web would be a huge job."
The real advantage of FreeNet is that data doesn't have to be artificially re-propagated. As a site is updated, users download fresh copies of each page, and these are inserted into FreeNet. Good site designers spend time figuring out how long their pages should reasonably stay in cache, and it wouldn't be too hard to extend the behavior of the WFW to take this into account. It would mainly depend on how you constructed FreeNet keys for each web page. Dynamic content is no different, as we're only dealing with the data after it has been inserted into the browser cache.
Thanks to everyone who has commented. I hope you'll all join our Sourceforge-powered mailing list (lists.sourceforge.net/...).
cach now!
dont' forget the Cache Now (electraglide.geog.unsw...) project.. they'v got some interesting ideas as well..
It could work
There are a couple issues about a distributed web; the primary one is security. How do you know if the data you are receiving from a web site is the actual data that the site operators put there? You don't really know. The thing is that it is not a simple task to impersonate a web server; so these sorts of things don't happen often. But in the case of a distributed browsing network, it becomes much easier to change the data on the fly. Users would need a way to ensure that the data they are receiving is authentic.
There are, of course, encryption schemes that would allow this, such as PGP/GPG-signed content. However, the problem then remains about key distribution.
Finally, you would have to also add capabilities to ensure that truly dynamic content is never replicated; Lots of web sites use cookies and other means to personalize the information shown, and it is important that this information not be shown to the wrong users.
If you think about it, a distributed network is truly the original goal of the Internet; it would totally decentralize the content making it very hard to destroy or shut down.
Documents, etc. Not up-to-date multimedia
Information that is useful both now and in a years time should be the main use of such a system. Small-time web pages put up for the sake of demonstrating something are the usual victims of the slashdot effect, and they are the
ones who would gain the most for such a system. A freenet-style system could/would make a valuable COMPLEMENT to the current web. Content more suitable for a freenet system would end up using the freenet system and content which is unsuitable would remain the way it is.
Nevertheless, I don't think that the browser should carry the weight of the logic. Basically, a mini-freenet node program/daemon should do that, and there should be a simple interface added to mozilla in order to connect with it. Given that, you could easily use such a system from mozilla but would neither be forced to use it where it was inappropriate, nor force to use it at all.
That said, a demand driven decentralised information system is something the the internet does not have, but could make good use of. That said, public key authentication needs to be more widespread --- since it would be necessary for many things to be authenticated (e.g. software releases could have a code/version number encrypted via a private key -- the software packages could then be easily authenticated, though this would require a little scrutiny in the design stages...)
Could this replace traditional file servers?
It is my goal to someday build a network for a company that consists of thousands of machines on everyones desks. The machines would run distributed applications, have distributed authentication, and have a distributed filesystem. In other words there would be no central server.
Freenet looks like the beginnings of a very good distributed filesystem. It allows the data to migrate towards those computers that are most interested in it. Add in some forced redundancy, access permissions, and a method to backup the data and it would ideal for a huge company or public institutions to allow them to store and share data.
Yeah!
[I'm one of the developers.]
Mozilla integration is something that has been talked about on the mailing list quite a bit. We're pretty much all for it. We haven't done any development in this area because the core developers have their hands full and no one else has volunteered to work on this.
Updating is an issue. But it's a minor issue. For dynamically generated pages, just don't cache them in Freenet. For pages which may change occasionally, updating in Freenet is necessary. It's on our todo list. It will be done. It's just a matter of choosing the right method.
Mozilla integration adds two things: robust, anti-slashdot-effect properties for the web and the ability to easily deal with various mimetypes and hyperlinks for Freenet. A freenet:text/README URL would be very convenient to have.
I am very excited about development being done on this project. I will be glad to help.
BTW, if anyone is at Defcon, I'm giving a talk about Freenet at 1:00 on Sunday, so drop by.
Privacy?
Just to go with a little knee-jerk reaction here... Should one worry about privacy in such a situation? I'm not sure I want other people to see what's in my web-browser cache (there was the time my sister-in-law sat down at my machine and started idly tapping the BACK button... *grumble*). You can tell where you're getting stuff from, after all.
Then again, I suppose it's not such a big deal since documents could have gotten into your cache from someone else's, based on how FreeNet works.
static vs. dynamic html
A major problem with caching is that it is a pain in
the butt for a site author to indicate static vs.
dynamic content. Proper caching signals go in http
fields via the server, not in html. Squid, et. al
have to make their best guess, and banner advertisers and
site-authors deliberately confound caches.
What would make caching easier, including this proposal,
is to augment the browser protocol. We have http:
for regular http, and https: for secure http, why not
add httpc: for cachable http.
The httpc: prefix would mean: "This is the official
URL for the document, but it is static so I don't care
where you get it."
Site authors can then easily distinguish static and dynamic
content. Many objects can be made static by adding
version numbers to them. If they update the object, they
update the version number. Advertisers would be able
to keep their banners dynamic.
Browsers can also be smarter about which methods to try
on a document. Internal, squid, and other proxy caches
would all benefit, and more importantly, automatic
mirroring solutions, like the proposed freenet solution,
would benefit.
I'd love to see all the "choose from amoung these 75
mirrors" pages disappear. This pollutes the namespace
with multiple copies of the same file under different URLs.
Some problems with "shared caches"
It seems people are overlooking a few problems. The static vs dynamic issue will
not get worse. Most large ISP's already do transparent proxying, where this issue
comes up as well. However, they tend to (and should) not cache script stuff. I,
for one, would not like to share my online banking cache results :) So there
should be ways to make sure some browing data does NOT get into the shared
cache. One step might be to never put SSL stuff in there, but wether people like
it or not, with all the government rules abroad and abound, the net is going to
become crypto very fast, defeating a lot of these strategies. And giving the user
an option to share or not share certain sites is just what you do not want to
burden the average enduser with, because they'll just share nothing (or that option
has to be the default from a security point of view).
Second, I'm not sure how the Freenet protocol works exactly, but I wouldn't like
others to ask my freenet client/ shared cache exclusively. That will allow them to
build up my user profile. Right now, we have enough cookies and 1 pixel banners
to worry about, we don't need yet another privacy invasion.
To come back to the slashdot effect, one poster had a good remark. Let slashdot
ask a site before posting wether or not they came suck a static copy of the content
and offer a "local mirror" option on their site (that might even hook to become
default as slashdot itself notices the remote site has gone done). Though I think
this would add quite a load on the slashdot servers, esp. when posting about
small sites (I think they might actually already at times not post a 'funny link' to
a site just to prevent the site from being wrecked).
P.
on the popularity idea
okay, so we have here a system by which, due to its nature, oft-requested files and information become more abundant on the network.
however, last I checked, computers do not have infinite storage, so the unpopular marginalized information suddenly becomes even more scarce than it was originally -- simply because it will be, if I understood all that right, topologically excluded in inverse proportion to how popular its files are.
so after a while it will be very easy to find britney spears and kid rock mp3s, but very difficult to find [name your favorite artist] mp3s.
the problem is that popularity is defined by central media. slashdot decides (effectively) what is important in the geek world. they post a url and everybody goes. now they post a url and it's copied all over the place, at the expense (due to finite storage) of other pages.
oh sure, the original webpage or server holding the unpopular file will still exist, but as the document said, the sorts of servers that "aren't useful" to the network will be pushed toward the periphery and get harder to access. **files that are on fewer machines than others will always be statistically harder to access**, simply because of the problematic nature of networks. shit goes down, crashes, is attacked, whatever. the more copies you have out there, the safer that information is. how many times have you tried to find out more about some program or project or anything, only to find that the only page in the world on which that information existed is now gone forever?
(if somebody could get me more information on the linux DIME project I would be eternally grateful)so you have all these nodes basically working to oust and keep marginal those servers which don't "play the game" and provide for the "useful" files. or are just down a lot and so aren't successfully sending files.
so, thanks to mtv and a popularity based file system, i will probably never hear my favorite music again. yay.
but this had to happen sometime. the net is just too damned big, and out of convenience we gravitate toward centralized sources to handle the information for us. slashdot, cnn, ftp.cdrom.com. this has all happened once before. it's called "the real world". people are so unpleased with the record companies and media and... everything. but it's happening on the net and is inevitable. mp3.com is a good example. most of their music sucks donkey balls right now, but, again, due to convenience, they have centralized a lot of their information -- you can find the most downloaded artists on their front pages and skip the search, ignore all the cruft, and find the good stuff. eventually they (or something like them) will be the first of the new breed of record companies. freedom for music, democracy, etc etc. all that crap you hear about the net, it's not true, and becoming less true every day. because we are lazy and like things fed to us.
to all you optimists out there who imagine that the net is really going to change the world, I hope I'm still around in 25 years to see where it all went. but now I just say, analyze the consequences of your seemingly simple actions on the net, and ask yourself "what if everybody is doing this?" because they probably are.
we have quite a big mirroring project going on right now: every day the net looks a little more like the world they show on tv and in the papers, a world where paved roads decide where we're allowed to go.
Get Cidera-ized!
Ironically, to get Akamaized would require being Cidera-ized first! Cidera provides the actual hardware/services that Akamai uses for its streaming products.
I'd have to disagree that the future IS or SHOULD BE localized cache sharing. Cidera already specializes in providing a satellite-based peering cache feed to ISP web caches and usenet feeds. This places the centralization at your ISP's POP (where it should be) instead of at your home, where there is always going to be a discrimination between your connection and the rest of the "Freenet".
META information
It seems to me, that the problem can be fixed with HTTP and some XML-Tags (or HTML-META-Tags). What is needed is the information:
What mirrors do exist
What mirrors should be used
On the other hand there is the What does the browser do with the information ?-problem. We had Mosaic that interpreted the LINK-element - others did not. The best way should be the development of a tag-set that could also be integrated in XHTML - I suppose there is one allready that gives the possibility to decide what do to next after the header of a document is retrieved. It is also possible to distinguish between stable and dynamic parts of a page (maybe only ads are dynamic). I don't think we need another new software - we need widely accepted protocols.
I like the description of the browser site in the article (like the "What's related") But I think buttons and menus should be more flexible (functions should be loadable dynamically - not build-in!) I see that the classical web-page comes to an end. Why shouldn't every web-page should have one pull-down-menu for it's contents? If an index is recognized by the browser he could display ist like he or the user wants. This would result in compatibility to every browser-type (even Lynx).
XML was invented because one would not implement new features and elements every time someone has an idea. It is time for the WWW, that this ideas get supported more effectively. And this will also mean: Not limited to BROWSERS at all!
Standard Caching problems...
The problems of dynamic distributed caching are pretty clear;
security: how does a client know it's from the desired source, and how does a source know that only the desired client(s) got it. I'd suggest PGP/SSL type signatures and encryption. Be aware that data targeted for specific client/s cannot be cached, except specifically for those recipient/s.
freshness: large amounts of (all?) content is dynamic to some degree. How do you avoid serving stale content, while still caching? http already supports object meta-data in the from of Expires headers and also supports freshness-checking with If-Modified-Since (IMS) requests. The problem is, how do you know in advance when the the data might have changed by (time to do an IMS request), or even better should definitely have changed by (no point in caching anymore)?
storage: with limited distributed storage space, how do you optimise hitrates. This requires optimising distribution of data and determining what data to actually cache. Throwing away suspected stale stuff will free up space, but might miss out on IMS hits. Keeping stuff that is fresh is a waste if no one is going to request it.
bandwidth: huge amounts of storage are useless without the bandwidth needed to get at it. How do you use that bandwidth effectively? Do you really want someone on the other side of the globe to be fetching 16M objects from your little part of the distributed cache over your 28.8k modem? Would they even want to? This is possibly the biggest problem with distributed, client-based caches; servers have all the bandwidth, so why fetch from a low-bandwidth cache? Putting large caches at high-bandwidth distribution points is nearly always more effective than having low-bandwidth clients share caches.
These requirements all interact and contradict with each other. It's a big juggling act. I'd have to say http is pretty comprehensive in its attempts at ensuring at least freshness(if very ad-hock in design), unfortunately not everyone is using it to its full capability.
The most exciting solution for bandwidth I've seen is rproxy
, which uses the rsync algorithm for doing delta-based updates to stale cached objects. This could replace IMS fetches entirely, and reduce bandwidth significantly on misses, even for totaly dynamic content.
Freenet is a cool idea, but I see its primary aim as dynamicly distributing data to protect it from censorship. Efficient distribution of data is one of the technical problems that must be overcome to meet that aim. I'd be quite surprised (and pleased) if they manage to do this so affectively that it becomes a general solution to data distribution problems outside their primary objective.
Bandwidth Problem
One of the posters mentioned a problem with bandwidth... such as do you really want to download a 16MB file from some guy with a 28.8 connection? Well... I'm not a programmer, but I am an engineer... If you know who has the file, couldn't you simultaneously fetch portions of it from multiple sources? In this case, the more available sources, the easier -- your end -- of the pipe saturates... in this case a good thing.
-- Phenym
AOLers already are using proxy caches
Those downplaying the usefulness of proxy caches erred in using AOLers as an example. AOL uses proxy caches now and has for a long time.
Dynamic content can be cached reasonably with an Expires or Last-Modified header. Cache-busting is the main reason for Squid and other proxy caches not being more popular than they are.
Re: Bandwidth Problem
> One of the posters mentioned a problem
> with bandwidth... such as do you really
> want to download a 16MB file from some
> guy with a 28.8 connection? Well... I'm
> not a programmer, but I am an
> engineer... If you know who has the
> file, couldn't you simultaneously fetch
> portions of it from multiple sources?
> In this case, the more available
> sources, the easier -- your end -- of
> the pipe saturates... in this case a
> good thing.
>
> -- Phenym
Hmmm... interesting idea. I suspect that the overheads in co-ordinating the partial downloads from all-over the place might kill the idea, but it's worth exploring...