Sharing Large Data Files

Click here to receive your FREE subscription to Campus Technology

Home > Sharing Large Data Files

Current News

Sharing Large Data Files

10/29/2003

By Terry CalhounBy , Joe St Sauver

J'e's back! While away, he had an aaaHHHAAA!!! moment, similar to his first browsing experience, and thinks he may just have spotted the "Internet2 Killer App." Read on to see why.

------------------------------------------

by J'e St. Sauver
University of Oregon Computing Center

This year's Fall Internet2 Member Meeting took place in Indianapolis, from October 12th to 17th. Besides being a nice opportunity to learn what's been going on in the Internet2 community (while also providing a chance to hash out issues with colleagues from other I2 schools face-to-face over a beer and a bowl of Cincinnati-style 3-, 4-, or 5-way chili), the I2 Member Meeting included the widely overlooked announcement of what may very well be the long-awaited "Internet2 killer app," a program from the University of Tennessee Knoxville Computer Science Department that g'es by the somewhat odd name of LoRS, part of the LoCI project.

Watching the LoRS demo at the Indy I2 meeting gave me the same sort of "aaaHHHAAA!!!" moment that I recall from when I first saw someone use an early version of Netscape to access a simple Web page: clearly, here was something that's going to profoundly change the way we do things online.

If you happen to have attended the LoRS session like I did, then you had had the chance to see an application that satisfies a fundamental need, much in the way that e-mail or the Web d'es. The need that LoRS satisfies is the need to be able to efficiently distribute large files, files that are too big to conveniently send by e-mail, files too big to conveniently download via the Web. You know the sort of files I'm talking about - large multi-gigabyte (or even multi-terabyte) experimental physics datasets, or CD-sized Linux ISOs, or those wonderful multi-hundred megabyte PowerPoint marketing presentations we all so love.

Yes, I know: people currently do move large (if not huge) files all the time via ftp, or chopped into digestible chunks via e-mail, or via Web pages.

Unfortunately, when folks move files using traditional tools, they don't tend to get very good network throughput, even over well-engineered, high-capacity, lightly loaded networks like Internet2's Abilene. (For example, the median throughput on Abilene for bulk file transfers is still less than 2.5Mbps. See Table 1 in the I2 weekly NetFlow report . One reason you're not seeing experimental physicists with fast Ethernet connections routinely saturating 100Mbps links is no mystery: it is simply a manifestation of our old friend, the TCP bandwidth delay product and its negative impact on untuned single-threaded network application throughput. (For a nice discussion of this, see: http://www.psc.edu/networking/perf_tune.html).

Serving large files from a single location, or even from a comparatively small set of distributed mirrors, also d'esn't scale very well. Ask anyone who hosts a Linux distribution mirror what they see when a Linux distributor kicks a new release out the door!