Articles / Client As Server: The New M…

Client As Server: The New Model

David Weekly writes: "A new model is emerging from the Internet. It represents the culmination of years of incremental evolution in the structure of the network and the clients that feed upon it. It is based upon the same principles upon which the Internet was founded. It is this: the client is the server."

darpanet

The Internet was created as a distributed network. Originally conceived as the Defense Advanced Research Projects Agency's network (DARPANET), it was to be able to withstand a nuclear attack from the Russians. There could be no single point of failure in the system, and in this was it had to be different from most other networks yet conceived.

People had previously grown used to the notion that there must be one central arbiter that oversees all transactions on a network: a Mainframe. This model has an obvious weakness: when the Mainframe goes down, the whole system is unusable. Then again, if there is only a single important point of failure, you could pay some people a lot of money to sit there and fix problems as soon as they happen (and hopefully insure that the problems never happen in the first place). Unfortunately, it's difficult to do this with regards to a nuclear bomb, so a different model was needed.

DARPANET provided this model by removing the server. It's like a model in which everyone hands mail to a friend who passes it to a friend who passes it to the recipient. While at first this might seem a little odd or inefficient, it means that it would be a lot harder for someone to stop the flow of mail to you (or the flow of mail in general). Instead of simply bombing the post office, now they have to assassinate each and every one of my friends to prevent me from getting mail. Going back to the real world, there would be no single point of failure which the Russians could bomb to take down our communications.

It was a revolutionary, strange way of thinking about things. To this day, some people don't understand it and ask questions like "Where is the server that runs the Internet?" or even "Where is the Internet?" It's hard to understand that every server on the Internet is a part of the Internet.

availability

These days, we are amidst an equally paradigmatic change which almost perfectly mirrors the first. Corporate servers, which distribute information and services to clients and participate in "e-business", need to not crash. Companies like eBay whose computers crash often get a bad name and lose billions of dollars on their valuations, almost a worse fate than the actual millions of dollars in customer transactions that go out the door when servers die.

A quick fix is to employ a large number of servers configured exactly the same way, so that if one goes down, traffic is quickly diverted to the others. Work is equally distributed amongst these servers by use of a "load balancer". This solves a few problems, but what if your server cluster is in California and the network link from California to New Zealand is getting bogged down? While the long-term answer is to invest in a faster connection to New Zealand, the short-term way to solve this problem is to put a server cluster in New Zealand. This sort of rapid expansion can quickly get expensive to deploy and manage. Some bright kids from MIT figured this out a few years ago and cobbled together what is now one of the fastest-growing companies out there: Akamai. (Hawaiian for "cool", if you're wondering.)

Akamai has already gone through the trouble of buying several thousand servers and putting them in network closets all around the world. The idea is that you can spoon off delivery of the parts of your site that don't change much (the pictures, the movies, etc.) to Akamai, and they'll take care of making sure that your readership can always quickly access your content. Cute idea. "Cool," even.

Distributed services lead to higher data availability. The more machines that are distributing your content in the more places, the more people will be able to access your content quickly. It's a straightforward idea. This notion of distributing work is also useful for distributing computation...

processing power

It's expensive to build really fast chips. It's really expensive. To make a chip twice as fast as today's high-end consumer chips costs about ten times as much money. That's largely because today's consumer chips are, pretty much by definition, as fast as it is possible to make them and have them still be reasonably cheap. If this wasn't the case, another company would have come along and made reasonably cheap, screaming fast processors and swept the market away. But the tight competition in the chip manufacturing business has kept the "bang for the buck" ratio screaming up through the ceiling, much to the delight of consumers.

It's important to note that making fast chips is expensive, because if I want ten times the processing power that comes in a top-of-the-line consumer PC, the best way to do that and save money is not to buy a machine that's ten times faster, it's to buy ten top-of-the-line consumer PCs. People have understood this general concept for a long, long time: wire together a bunch of processors to get a very, very fast machine. It's called "Massive MultiProcessing" (MMP) and is pretty much how all of the supercomputers of yore (and of today!) work.

The recent concept is that it's possible to do this with off-the-shelf PCs. Recently, software (such as Beowulf) has been developed to make it very easy to make a cluster of PCs act like one very fast PC. Sites that previously deployed very expensive custom supercomputer systems are actively investigating using massively distributed commodity hardware to serve their computing needs. That would be remarkable as-is, but this concept of distributing computing cycles has gone even farther than clumps of commodity hardware: it's gone into the home.

seti @ home

For roughly the last forty years, there has been a serious and conscientious effort to search for intelligent life in the universe by listening for patterns in radio transmissions. The process of analyzing incoming radio transmissions for patterns such as those that might be emitted by intelligent life forms is mind bogglingly difficult, requiring vast numbers of computations. While privately funded, the members of the Search for ExtraTerrestrial Intelligence project didn't have enough money to process all of the data that they were getting. They did, however, have a sizeable fan base. (A number of people on this planet think it would be pretty cool/important to discover intelligent life out there.) So what did they do? They distributed the work.

Some clever programmers put together the software used for analyzing the data returned by the Arecibo antenna (the largest radio receiver on Earth), put some pretty graphics on it, got it to act as a screensaver, and put it on the Web. Several hundred thousand people downloaded it and ran it. While they're away from their computers, this pretty screensaver crunches through vast quantities of data, searching for patterns in the signals. The SETI project (as of this writing) in this way has a "virtual computer" that is computing 13.5 trillion floating-point operations per second.

(I feel I should also mention distributed.net, which spends its time using people's computing power to crack cryptography codes. Their "virtual computer" is currently cracking a 64-bit cipher known as RC4 at the rate of 130 billion keys per second.)

data services

So it's now clear that it's advantageous to distribute computation and the serving of data across as many computers as possible. We've seen how a few projects have distributed computation across end users, but what projects have distributed data services?

Napster is one of the first and best examples of end-users acting as distributed servers. When you install Napster, it asks you where your MP3 files are. You tell it, and it makes a list of what MP3 files you have, how long each song is, and of what quality the recording is. It then uploads this list (but not the songs) to a central server. In this way, the central server has a whole bunch of lists. It knows who has what music, and you can ask the server who has songs by Nirvana and then contact those other users (while your Beck tunes are possibly getting served to some Scandinavian with a predilection for American music). This model allows information (in this case, MP3 files) to be rapidly and efficiently served to thousands of users.

The problem with it is both technical and legal. There is a single point of failure: Napster's servers. While there is more than one server (the client asks a "meta-server" what server it should connect to), they are all owned by Napster. These servers, unfortunately, do not share their file lists between themselves, and as a result, you can only share files (and see the files of) others connected to the same server that you happen to have connected to. Napster is currently being sued by the RIAA for acting as a medium for distributing illegal MP3 files. While it is true that Napster can be easily used for illegally distributing MP3 files, they themselves don't actually copy the bits for users; it's more like acting as a Kinko's that happens to be used by subversives than actually distributing copies of MP3.

If you are a Napster user, you should be worried about this lawsuit, because if the RIAA succeeds, they will probably want to shut down Napster's servers, thus theoretically shutting down the whole Napster network. In short order, they could quickly close down any Napster clones because of the legal precedent that the anti-Napster case would set. Boom. Game over, no more illegal music.

Theoretically.

a virtual internet

The RIAA mentality is one and the same as that of the Russians of yesteryear: a desire to stop the flow of information through the network. The answer to the Russians is one and the same as the answer to the RIAA: a completely distributed system. If every client on the network was connected to a handful of other clients, each of which in turn connected to others like some apocalyptically enormous online incarnation of Amway, then every person could have some connection to every other person through a chain of mutual acquaintances. It's Six Degrees of Freedom. (There exists a theory that says that, on average, you know someone who knows someone who knows someone who knows someone who knows someone who knows anyone in the world. That is to say, you are about six degrees from every human on the planet.)

This is a "virtual Internet" of sorts in which links are not physical (a wire from you to me) but logical (I know you). Data flows through this "web of friendship" in such a way that it looks like you are only talking with your friends, when really you are talking to your friends' friends, and so forth.

gnutella

The same rebellious college hacker genius who created the fabulously popular MP3 player "WinAMP" (and was subsequently bought out by America Online, now AOL-Time/Warner-Netscape-EMI-And-Everything-Else-But-Microsoft) happily hacked out a program that enables the free exchange of just about any kind of file over such a peered network. Unfortunately, his bosses discovered it halfway through development and quietly tried to erase the fact that it had ever existed in the first place. The name of the program? Gnutella. (Named after the delicious chocolate spread Nutella.)

Since there's no central server around which Gnutella revolves, AOL's shutdown of the project didn't actually stop Gnutella from working. A growing user base of several thousand souls (myself included) uses the product on a daily basis to share files of all types, from music to movies to programs. At last check, there were about 2,200 people using it, sharing 1.5 terabytes of information. Wow.

There's no way to shut it down. There is no organization to sue to stop it. There is no server to unplug that would bring the network tumbling down. As long as at least two people are running the software, the network is up and running.

freenet

There are even more advanced projects in the works that will build upon these notions to create an even more powerful incarnation of a peered network that incorporates notions of perfect anonymity, trust, secrecy, realtime communication, and even banking. Freenet is perhaps the furthest along in this, although it has a very long way to go as of this writing. If you're interested, you can read about my own scheme for a Secure + Anonymous File EXchange.

the future net

Akamai has shown that it is clearly advantageous to have content distributed by as many nodes as possible. Companies are willing to pay good money to have their content on thousands of servers all over the world. Gnutella is showing that it is possible to create distributed networks that cannot be shutdown, even in the face of legal and technical opponents. Napster shows that such networks can become popular and that people are willing to act as servers. Seti@Home shows that people will even allow others to use their computing power for a "greater good."

What is enabling this now? Well, computers are, unsurprisingly, getting faster every year. The average desktop that's sold to Joe User for doing word processing, email, and Web browsing can, when properly configured, deliver hundreds of thousands of email messages a day, serve millions of Web pages, route Internet traffic for tens of thousands of users, or serve gigabytes of files a day. (Joe probably isn't aware of this and will still kick it when Word takes five minutes to load.) His hard drive could store 100,000 Web sites each having ten or so pages, email for 1,000 users, and a few thousand of his favorite songs. Furthermore, if Joe has DSL or a cable line, he's got a static IP (an address on the Internet that doesn't change often, if at all), is almost always connected to the Internet, and is online at high speed.

In short, Average Joe's computer resembles one of the best Internet servers of yesteryear.

If thousands of Joes end up running "community" applications like Gnutella, they can take advantage of their connectivity, disk space, and computing power. New "co-hosting" services will spring up like popcorn in the microwave. Here are a few possibilities in that direction:

the distributed future

Visualize, for a moment, sending your Web site into a collective ether, to be served by hosts around the world. If one computer goes down, others will spring up to serve it. Your page never goes down. Your friends send you email encrypted so only you can read it, and it is stored on half a dozen of your friends' computers, accessible to you from anywhere on the planet. All of this in exchange for setting aside a small chunk of your hard drive (100 megabytes or so) and a little bit of your bandwidth to serve Web pages and people's email. Any content that you consume (except for your personal email!) is instantly rebroadcast over the network. Your computer helps data flow to where it is popular.

requiem server

In the future, there will be no need for centralized services. All content will be available on this peered network. Strategies for "partial consumption" such as letting people read the first few paragraphs and charging for the story or hearing the low-quality song and charging for the audiophile version will be adopted along with anonymous payment schemes. It will be possible to send intelligent agents to this network to search for books, music, or other merchandise. Clients (such as Amazon.com, CDNow, eBay, your neighbor, etc.) that have a match for the merchandise will communicate with you through the peered network, preserving your anonymity. You will be able to make an anonymous payment (or merely a secure payment if you prefer) and your goods will be on their way to you. No more URLs. No more servers that crash, email that is unavailable, Web sites that you can't get to, or data that you can't find. It will truly be the end of the server as the line between what it means to be a "client" and a "server" on the network becoming increasingly blurred to the point of indistinguishability.

a footnote on wireless

It is worth pointing out that wireless Internet access may well become democratized as well. Since high-speed wireless Internet is taking so long to reach America, citizens may just do it themselves. Apple recently popularized the IEEE 802.11 standard for wireless Ethernet by including AirPort in the iBook. Wireless Ethernet cards are now available for PC and Mac desktops and portables. More exciting yet, people have been working on extending the range of the AirPort from a couple hundred feet to tens of miles. Imagine a future a few years away in which one person every city block has a base station and everyone else jacks in. Some have postulated this as being the ultimate incarnation of the communality and free spirit of the Internet.

David Weekly is a senior majoring in Computer Science at Stanford University. A programmer since the age of 5 and a veteran of the MP3 scene, he's working on graduating in June and generally figuring life out. He offers a tip of the hat to Kevin Doran for inspiring him to write these ideas down.


T-Shirts and Fame!

We're eager to find people interested in writing editorials on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an editorial gets a freshmeat t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.

Recent comments

24 May 2000 14:32 Avatar toddboyle

Magical transaction fabric in the sky
I'm an independent web accounting consultant and evangelist. I maintain a free info. website you might enjoy. David Weekly is God! I want you to build an ecommerce platform based on freenet.

Basically, individuals and small business will send and receive invoices, orders and payments over the internet instead of printing and mailing. The question is yours: will this be geodesic or will we be paying rents to hubs and gatekeepers the next fifty years, like we are with the telcos and the banks?

Based on about 5000 hours of research I think the webledgers will be so numerous and functional, the fees and rents will be low, and today's hubs, banks etc. will be a laughingstock, in comparison. If you're curious here is a list of webledgers and other BSPs http://www.gldialtone.com/links.htm

But every ecommerce platform and webledger, today, and on the drawing board is still a book-entry system on a server someplace, (except some of the digital bearer cash discussed on www.philodox.com DBS mailing list)

The ideal ecommerce environment would be a "magical transaction fabric" in the sky, where we will find bids and offers with zero friction, conduct business in utter safety and privacy, and where all transaction data would be stored with total reliability and redundancy but decentrallized, with the encrypted bits dispersed among countless trillions of other transactions and millions of servers, impossible for anybody except the principle parties to retrieve, and impossible to lose, yet retrievable with your key anytime.

(translation: I have discoverd http://freenet.sourceforge.net/ (grin))

"it" is going to need some resource management mechanism.

When you start using the magical transaction fabric, the data sizes and bandwidth assigned to your computer by the fabric would be stored based on the data sizes and bandwidth you are loading onto the fabric with your business. If you unplug your computer the fabric would detect it and start repairing the lost hash by shovelling it out to nearby nodes. The fabric would punish you by increasing your datasizes and bandwidth next time you login, eventually killing you if you keep doing it.

Todd http://www.GLDialtone.com/webledger.htm

19 Apr 2000 02:11 Avatar mrshiny

legality issues
In regards to some people's view that Napster and Gnutella are illegal because they are used for piracy purposes, I just have to say that it doesn't matter if no one distribuites shareware or source code on Gnutella and you've never searched for original artists on Napster. That wasn't my point at all; the point was that you can't just decide that a program is illegal because people use it to do bad things. That kind of reasoning is flawed; it's like saying that netscape shouldn't have ftp support, because the only people who use ftp in windows are downloading warez. I don't know many people who run windows only, and use ftp who actually download legal files. Yet if I claimed that ftp support in windows equals piracy, people would say that it was stupid. The point is that a data transfer protocol is not illegal. Do you know anyone who actually made legal copies of things with cd burners? Everyone I know who has one burns pirated CD software; virtually no one uses it as backup or (legal) software distribution (I'm not including people in industry, only home users). Are CD burners illegal? Of course not. Photocopiers can be used to make unlimited numbers of copies of copyrighted works; are they illegal? Nope. The fact is that just because a tool can be abused is no reason to make that tool illegal.

Furthermore, the claim that NFS/SMB are "enough" to share data with the rest of the world, or that email is sufficient, is garbage. Email is not designed for file transfer; that is a bad hack that should be discouraged. That's why most ISPs bounce emails larger than a few MB. And NFS and SMB are not well suited to Internet use. For one thing, they require a lot of configuration, and provide too little security. For another, they are not suited to the needs of people with dynamic IPs and groups that are not fixed in size or membership. They just don't provide the same functionality that Napster or Gnutella provide.

Now, I'm not condonning piracy; nor do I claim that Napster or Gnutella have any amount of security. But they were designed because there was no such tool that provided this functionality. It doesn't matter if the inventors had good or bad intentions; it doesn't matter if the tools are used for illegal purposes. The point is that a tool is not intrinsicly bad, and saying that "People share warez on Gnutella so Gnutella is illegal" is just as ludicrous as saying that "Cars can be used to kill people so cars should be illegal".

18 Apr 2000 11:04 Avatar drwilco

Security and Trust
The problem I see with such a widely distributed network is security and trust. For instance with the mail example. If 10 friends all store the same email to addressed to me, there's a good chance I would get that email, even if a few of their machines are down. But what if a fwe of them get together and forge an email. How would I know it's not the real thing?

This is just an example and a very simple one. Considering SMTP right now is also very open. But look at email-SPAM. As someone else noted, that would be very easy to do with Gnutella.

Without a central authority it gets really hard to implement a model based on trust that can span the entire (or larger part of the) internet.

One could try with Certificates and Public Key concepts, but still, one would have to say "I trust that person". And can you really trust all your friends.

And how far would you let that trust carry. If you trust person A, and person A trusts person B, does that mean you trust person B? It would seem the right approach for a Akamai like concept, but what about the "7 degrees of seperation" theory? I'm not completely sure, but I believe the theory says that between any individual on earth and any other individual on earth there are only 7 people. Frankly that has me worried. It might not be 7, but anything higher than 10 seems a bit too much. That's 10 people. So if the trust goes 11 levels deep, I trust everybody on earth. Including crazy virus writers, and childporn collectors.

So although "The client is the server" seems like a cool idea, I think the whole trust&security aspect of it should be examined and examined really carefully, because frankly I think that the scenario of introducing bogus files and noise that was posted earlier is a real threat.

17 Apr 2000 18:10 Avatar dmitrikondratiev

The New Model and Russian Threat
After reading your "Client As Server: The New Model" I'd like to draw your attention to the fact that today is April of year 2000 which you probably forgot when using "Russians" as an example of universal threat to humanity in your text.
This loss of memory probably happens when coding or writing articles around the clock.
On the technical side I agree with your view on distributed computing being myself in network programming for about 15 years already. I was lucky to take part in the architecture and actual development of the first Russian heterogeneous LAN ALISA in the early 80th even before the Novel Netware come into existence. ALISA has been successfully deployed on about 1000 sites in many parts of the world (including FTP Software where some of ALISA developers work since then) and allowed to interconnect DEC PDP, IBM mainframe and PCs into network providing virtual devices such as terminals, tapes, disks and
printers.
Even that time, when working on ALISA I don't remember anybody around me in Moscow who would consider using nuclear weapons against any other country.
It was quite a discovery for me later when I was working in Silicon Valley in 1996, 1997 and occasionally had a small talk with a cop of the fact to what an extent most public in the States was zombied by Russian threat. This guy told he likes that we can just chat as normal people now. He also said that he was told many horror stories in school about possible Russian invasion.
Even policemen has come over this brain-washing at least three years ago !
The more sorrow I feel now in year 2000 when I come across mentioning Russia (actually your example is the first one in many years) as a threat to the world in the technical! article. This is ridiculous at best. Much more appropriate as an example of a real threat would be to use "Corporate America" which I got to know so well during two years I lived in the Valley.

Dima

17 Apr 2000 17:50 Avatar spotthedog

But gnutella has a serious flaw...
Try this experiment: create a few files full of garbage and put them in your gnutella outgoing directory with provocative file names, eg SexySlut.jpg or FatBoySlim_PraiseYou.mp3. Watch as people wasting their bandwidth keep downloading these files.

Of course this is not very nice behaviour, but it serves to illustrate a very real possible denial-of-service attack, albeit one characterised by gradual degradation.

Imagine you are the RIAA or the Chineese Government and wish to prevent the spead of the technology by making it (almost) useless. You can connect to any gnutella node and obtain a list of other nodes, and consequently walk through mapping the network as it morphs from moment to moment. Keep a connection open to every known node and each time a search is initiated, respond by saying you have a file which matches exactly what has been requested. You can even use AI learning techniques to optimise the construction of responses to make a best guess as to what will most likely result in a download. You can also keep changing your IP number to make it difficult for gnutella users to learn to avoid your 'noise'.

The problem is that gnutella as currently designed does not implement any quality control mechanism. The solution/challenge is to design such a mechanism that works in a self-organising, distributed network.

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.