ORNL Net100

Project ended 10/1/04.

The Net100 project is a DOE/MICS sponsored project to develop a network-aware operating system. The project seeks to measure and understand end-to-end network and application performance, and apply that knowledge to tuning networks and applications, eliminating the "wizard gap." The project is looking at both bulk transfer over high latency/bandwidth networks and grid-based applications, especially (but not limited to) DOE's ESnet. The major components are

• developing/deploying network probes and sensors to characterize the network and distributed applications
• Web100 kernel extensions to monitor and tune network applications
• TCP tuning

The project has some novel approaches, including

• Net100 leverages the Web100 kernel to passively collect data on individual TCP flows
• Net100 can tune several TCP parameters for designated flows at connection start and during the flow. Net100 can or will tune
  • TCP socket send/receive buffers
  • TCP AIMD parameters, the notion of a virtual MSS/MTU
  • DUP threshold to absorb re-ordering
  • Delayed ACK tuning on the receiver
  • slow-start mods (initial window size and initial ssthresh)
  • burst control
  • transport algorithm (e.g, TCP Vegas, or ?)
• Net100 will determine optimal paths and whether to use multiple streams and/or multiple paths

The ORNL Net100 team:

• Tom Dunigan
• Florence Fowler
• Nagi Rao
• Bill Wing
• Steven Carter

• PSC and the web100 team (Matt Mathis) and their TCP tuning guide
• Brian Tierney, LBNL and his netlogger and his TCP tuning guide and his net100 page and Jason's Net100 progress page

Recent Net100 accomplishments at ORNL

• Net100 added to SGI Altix; tunable TCP Vegas (9/03)
• UDP transport evaluation (Tsunami, FOBS, SABUL, RUDP) (9/03)
• Net100/Web100 invited to contribute to Linux 2.6 (8/03)
• Net100 added to Cray X1 network frontend, 4x speedup over WAN (8/03)
• experiments with Kelly's scalable TCP
• PAM 2003 paper Measuring End-to-End bandwidth with Iperf using Web100 (3/03)
• Vegas testing over atou and with Floyd slow start (9/02)
• use WAD or kernel to continuously tune AIMD values of a flow (see early results 7/02)
• use NTAF data from WAD to set buffer sizes (7/02)
• use WAD to tune single and parallel stream gridftp (6/02)
• experiments with Floyd's slow-start and AIMD mods with our TCP-over-UDP (atou) and with Linux kernel and WAD (6/02)
• SC 02 paper accepted, A TCP Tuning Daemon, Dunigan, Mathis (PSC), Tierney (LBNL)
• used wad to auto-tune buffer sizes and additive increase (virtual MSS) for TCP transfers between ORNL/NERSC/PSC, order of magnitude speedup (5/02)
• combined dynamic-right sizing with web100 in linux 2.4.16 (2/02) auto-tuning summary
• added netlogger instrumentation to webd (1/02)
• used wad to auto-tune buffer sizes TCP transfers between ORNL/NERSC (1/02)
• preliminary SCTP evaluation (1/16/02)
• Web100 kernel patches to add event notifcation (12/01, 2/02)
• used Web100 to manually tune an HSI/HPSS transfer from NERSC to ORNL at SC2001 (11/01)
• Java applet bandwidth tester using web100 server (10/01)
• a TCP over UDP (atou) test harness for link characterization and protocol testing (AIMD, virtual MSS, delayed ACK)
• also see our web100/net100 highlights a timeline of accomplishments to date

Ongoing work in 2004

• Net100 and Cray X1 and SGI Altix
• TCP Vegas and FAST evaluation
• TCP Westwood re-evaluation (part of linux 2.6)
• UDP transport evaluation (Tsunami, FOBS, SABUL, RUDP)
• design/implement WADlite

• NERSC/ORNL link characterization
• web100 webd and WAD
• virtual MSS and AIMD experiments (ns, atou, WAD)
• characterizing auto-tuning , Feng's DRS and Web100 autotuning (tests with NISTNet)
• tuning parallel streams
• evaluating HSI, pftp, bbftp, bbcp, gridftp
• evaluating path characterization tools (pipechar, pathprobe, pathrate)

From the proposal, you can views ORNL's Net100 milestones here. During the first year, ORNL's Net100 goals and progress

• Probe NERSC/ORNL bulk transfer study One of our first demonstrations of the Net100 tools and tunable TCP will be transfering large data sets between NERSC and ORNL. We are trying to characterize the ESnet links and the HPSS/Probe applications currently used to transport data and other potential transport services.

  Here are some ongoing activities and results of our work so far.
  • tcpdump/tcptrace analysis for HPSS data movers between NERSC/ORNL, see recent results
  • evaluate pftp, bbftp/bbcp, gridftp, HSI (on going)
  • Using various network tools and applications we have been trying to characterize the ESnet link between ORNL and NERSC and to understand TCP's behavior on this high delay/bandwidth path. Results from OC3 and OC12 studies to date.
  • test jumbo frames over ESnet (to do)
  • test ECN over ESnet (to do)
  • gather ESnet router data (to do)
  • add netlogger instrumenation (in progress)

• TCP optimization studies Without resorting to parallel streams and non-TCP transports, we seek to tune single TCP flows across high bandwidth and high delay networks.

  • we have been doing a literature review of TCP congestion management
  • we are evaluating TCP tuning options for various OS's (AIX, Linux)
  • we are evaluating auto-tuning performed by Feng/LANL, Mathis, and Linux 2.4, see auto-tuning summary
  • we have developed a TCP over UDP test harness, see atou page
  • determine what can be used for TCP tuning: window sizes, RTTs, losses, cwnd, ssthresh (in progress)
  • determine what can be tuned in TCP: window size, dup threshold, initial window, AIMD parameters, speculative recovery
  • we have done TCP simulations with ns
  • we have done emulations with NISTnet
  • Testing parallel transfers with iperf, bbftp, gridftp
  • Installed SCTP on two ORNL linux boxes, initial evaluation complete (summary)
  • added Vegas and D-SACK to atou (to do)
  • set up an emulator testbed, NISTnet -- done

• Network tool evaluation We are installing Web100 kernels and various network tools at various national labs and universities. These tools will be extended with Net100 and incorporated into the the projects NTAF.

  • We have deployed a GigE Linux/Web100 host at ORNL. We have a Web100 node at UT. We just established a GigE Web100 node at NERSC. We accounts on Web100 hosts at LBL, NCAR, NCSA, and SLAC. We have test nodes at SDSC and ANL.
  • We have instrumented ttcp and iperf with Web100, and have developed a Web100/java bandwidth tester, see web100 testing
  • we have added event notification to Web100
  • develop simple Web100 logging daemon webd and tracing daemon, traced, to save web100 info on various links, a pre-cursor to auto-tuning and a NAIB (in progress)
  • we are developing WAD, (workaround daemon) to tune a given TCP flow, see WAD results
  • We have been testing iperf, pathrate, pathprobe, netperf, pipechar, pchar, ttcp, and various tools of our own design.
  • install Netlogger and Enable (in progress)
  • evaluate NCS and NWS and NIMI
  • collect ESnet router data (maybe from MRTG hosts) (to do)

Interactions
The Net100 team is also working closely with the other DOE/MICS networking projects including
SciDAC projects (see networking projects)

• DOE Science Grid: Enabling and Deploying the SciDAC Collaboratory Software Environment
• A High-Performance Data Grid Toolkit: Enabling Technology for Wide Area Data-Intensive Applications
• INCITE: Edge-Based Traffic Processing and Service Inference for High Performance Networks
• Earth System Grid II: Turning Climate Datasets into Community Resources
• Optimizing Performance and Enhancing Functionality of Distributed Applications Using Logistical Networking
• SDM ISIC Scientific Data Management HPSS and Probe
• Bandwidth Estimation: Measurement Methodologies and Application
• Shedding New Light on Exploding Stars: Terascale Simulations of Neutrino-Driven SuperNovae and Their NucleoSynthesis

• Feng's High-Performance Transport Protocols
• Tierney's Self-configuring Networking Monitoring
• Schwan's IQEcho: Interoperability and Qaulity of Service
• NERSC/ORNL Probe project
• Preliminary discussions with IBM AIX TCP group in Austin and with IBM Linux, Net100 mods to Cray X1 network frontend. Looking at adding Net100 to SGI Altix.

Net100 presentations and papers
net100.org is the official web site, but here are relvant links (pardon the Microsoft bias)

• Net100 final report presentation 9/16/04
• Net100 final report 9/13/04
• Net100 project status report 2/27/04
• ORNL DOE/MICS briefing 1/8/04
• ORNL SC03 Net100 poster
• DOE/MICS Net100 summary report and power point 8/18/03
• ORNL seminar Net100 overview 8/14/03
• Network Magazine End-to-end Performance on the Internet 4/03
• ESnet ESCC Net100 status Miami, 2/4/03
• PAM2003 Measuring end-to-end bandwidth with Iperf using Web100 Tirumala, Cottrell, Dunigan, accepted 1/15/03
• SC2002 TCP Tuning Daemon talk (ppt) 11/19/02
• SC2002 ORNL Net100 overview graphic and Net100 fact sheet 10/30/02
• SC2002 paper, A TCP Tuning Daemon
• Net100 overview for PNNL, 10/21/02
• Rao/Feng EEE International Conference on Networks (ICN'02), "Performance Tradeoffs of TCP Adaptation Methods"
• ORNL Web100 activities 8/1/02
• ORNL Net100 status and 2nd year 7/31/02
• ESnet meeting ORNL Net100/WAD 4/9/02
• ORNL tech rpt A TCP-over-UDP Test Harness, ORNL/TM-2002/76
• PI meeting (access grid), ORNL status and Rao's status of statistical estimation/analysis work 3/26/02 and LBNL slides and psc slides
• Tempe Internet2 End-to-End Performance Initiative (E2Epi) Measurement Workshop, ORNL Net100 slides, 1/30/02
• quad chart 1/7/02 and ORNL Net100 summary for DC SciDAC PI meeting 1/13/02 and Ndousse's all DOE net projects
• my net100 slides as part of DOE/ORNL network workshop 11/30/01
• ORNL's SC2001 Net100 BOF slides 11/15/01
• ORNL's Denver Net100 slides 11/8/01
• press release, 9/26/01
• Web100 testing at ORNL, presented at Web100 conf., 7/24/01
• Net100 overview presented at Web100 conf., 7/24/01
• quad-slide summary, 9/11/01
• executive summary, 9/10/01
• proposal, 4/23/01 (private)

Related Work at ORNL
Here are some of ORNL's network performance projects from ORNL's Network Research group

• long delay, high bandwidth bulk transfers
• a TCP over UDP test harness
• Web100 experiments and Java applet bandwidth testers
• AIX TCP/UDP bugs we uncovered
• network infrastructure problems we uncovered
• low latency nets (cluster/parallel computing)
• adaptive/multipath routing with N. Rao
• ATM latency testing with GPS

Links

Visit the network performance links page for tools, RFC's and papers.