Faster Bulk Transfer Starring: *UDP*

Study and Evaluation of 3 UDP protocols

3.3. Fast Object-Based data transfer System(FOBS)
FOBS is an application library implemented in C and uses many of the NETBLT ideas but at the user level rather than the kernel level. The primary paper on FOBS, An Evaluation of Object-Based Data Transfers on High Performance Networks , states that by 'object-based' they mean that none of the data transferred can be assumed to be correct until all of the data has been received. The code seems to indicate that "all of the data" means a "CHUNK_SIZE" which is currently preset at 100 MB. The application does not have access to any of the data until the "CHUNK" is successfully received. That assumes that both sender and receiver have pre-allocated buffers large enough to accommodate the complete size of a "CHUNK". Each CHUNK of data is transmitted in segments--10000(or fewer depending on the amount of data remaining) packets carrying 1466 bytes of data. Loss information is sent from receiver to sender after a segment of data has been sent. FOBS presets a tolerated loss rate(1%) and makes the assumption that packet losses are not necessarily the result of congestion.

1. use of TCP/UDP channel(s)
FOBS uses two TCP channels to transfer control information between sender and receiver. One channel for ENDOFSEGMENT/DONE/FEEDBACK/ACK packets and one for COMPLETEDPKT/WRITECOMPLETEDPKT packets so that the receiver can signal the sender when all the data in the current CHUNK has been successfully received and written to the file. Setsockopt() is used to set all the sender's socket buffers(SO_SNDBUF , SO_RCVBUF) to socketBufferSize(100000). SO_REUSEADDR and TCP_NODELAY are set for all TCP sockets for both sender and receiver.
One UDP channel is used to transfer data from sender to receiver-- the UDP socket SO_SNDBUF was also set to socketBufferSize(100000) and SO_REUSEADDR is set for both sender and receiver. The UDP socket SO_RCVBUF did not appear to be set anywhere???
A timed select() is used to detect incoming packets on TCP and UDP sockets for both sender and receiver. 

TCP control packets from sender to receiver:
#define DONEPKT	          1 - sent after all original data or all
				retransmitted data in a CHUNK has been
				sent
#define ENDOFSEGMENTPKT   2 - sent at the end of each segment

TCP control packets from receiver to sender:
#define COMPLETEDPKT      1 - sent after all packets in a CHUNK have 
				been received
#define FEEDBACKPKT       2 - sent in response to the ENDOFSEGMENTPKT
				or DONEPKT--
			        when all packets in a segment have not
				been received
#define NACKPKT           3 - sent in response to a DONEPKT
#define ACKPKT            4 - sent in response to the ENDOFSEGMENTPKT
				or DONEPKT-- 
				when all packets in a segment have
				been received
#define STOPPKT           5 - not used
#define WRITECOMPLETEDPKT 6 - sent after a COMPLETEDPKT indicating that
				all data in the CHUNK has now been 
				written to the file

2. rate-control algorithm
As you start up FOBS, you are asked to enter the NIC speed for sender and also for receiver. The currentSendingRate starts out as a function of NIC speed. An acceptable loss rate(1%) is assumed. FOBS links rates calculated at initialization to a network state machine and adjusts the currentSendingRate to one of these precalculated rates based on feedback from the receiver as the flow progresses. The states are listed as GREEN, YELLOW and RED. Each state has a MaxRate, MinRate, MidRate, increasingStep and decreasingStep all calculated at sender initialization with the selected NIC speed and acceptable loss rate as inputs. The transfer starts out in the GREEN state using MaxRate. 
MaxRate[NS_GREEN]=nicRate;
MinRate[NS_GREEN]=0.85 * nicRate;
MidRate[NS_GREEN]=(MaxRate[NS_GREEN]+MinRate[NS_GREEN])/2;
increasingStep[NS_GREEN]=(MaxRate[NS_GREEN]-MinRate[NS_GREEN])/5;
decreasingStep[NS_GREEN]=(MaxRate[NS_GREEN]-MinRate[NS_GREEN])/5;
 
MaxRate[NS_YELLOW]=MinRate[NS_GREEN]-1;
MinRate[NS_YELLOW]=0.4 * nicRate;
MidRate[NS_YELLOW]=(MaxRate[NS_YELLOW]+MinRate[NS_YELLOW])/2;
increasingStep[NS_YELLOW]=(MaxRate[NS_YELLOW]-MinRate[NS_YELLOW])/10;
decreasingStep[NS_YELLOW]=(MaxRate[NS_YELLOW]-MinRate[NS_YELLOW])/7;
MaxRate[NS_RED] = MinRate[NS_YELLOW] - 1 ;
MinRate[NS_RED] = 0.02 * nicRate;
if(!MinRate[NS_RED])
        MinRate[NS_RED] = 1;
MidRate[NS_RED] = (MaxRate[NS_RED])  / 2;
increasingStep[NS_RED] = (MaxRate[NS_RED] - MinRate[NS_RED]) / 15;
decreasingStep[NS_RED] = (MaxRate[NS_RED] - MinRate[NS_RED]) / 5;
The currentLossPercent, calculated from the data sent by the receiver in response to an ENDOFSEGMENTPKT, is input to the following formula to determine the current available bandwidth. 
        availableBW = (currentRate - (((double)(currentLossPercent - acceptableLossRate)/100)* nicRate));
The availableBW is then compared to the previously calculated table to determine the current state of the network and, correspondingly, whether the newSendingRate should be increased or decreased from the currentSendingRate. The sender checks the actual sending rate against the desired sending rate after a burst of packets has been sent. Right now, burst size is set at 25. If the sending rate is greater than the currentSendingRate, a gettimeofday() calculation is done until time to send the next burst.

3. data sending algorithm
The sender sends a segment's worth of data packets and then sends an ENDOFSEGMENTPKT--or if this is the last segment of this CHUNK, sends a DONEPKT--requesting feedback from the receiver. After an entire CHUNK of new data has been sent, the sender resends missing packets again sending an ENDOFSEGMENTPKT packet after each segment's retransmissions. After all the segments in this CHUNK have been acked by the client, the sender receives a COMPLETEDPKT and then waits for a WRITECOMPLETEDPKT indicating that the receiver has received all data in the CHUNK and has finished writing it to the file. The sender then begins sending the next CHUNK of data.
In deciding what to send, packets that have been transmitted n times take precedence over the packets that have been transmitted n+1 times. A sending state machine is used to send out data and control packets and process feedback from the receiver.
  1. initialize
  2. get current time
  3. for each transferSize/CHUNK of data to be sent
  4. wait for the final WRITECOMPLETEDPKT from the receiver
  5. get current time for entire transfer
  6. do ending calculations and close down
4. data receiving algorithm
The receiver receives and processes control packets first. When an ENDOFSEGMENTPKT or DONEPKT is received, a FEEDBACKPKT is sent with a list of packets missing from the current segment or, if all packets have been received for that segment, an ACKPKT is sent. After all packets for the current CHUNK have been received, a COMPLETEDPKT is sent, the buffer is written to the file, and a WRITECOMPLETEDPKT is sent.
  1. initialize
  2. for each transferSize/CHUNK of data to be received
  3. print statistics and close down
Graphs illustrating a representative data transfer over NistNet using FOBS may be found here.

5. unique features
***sending algorithm
Data that has been sent n times has preference over data that has been sent n+1 times. Similar to a version of NETBLT, FOBS is lock-step--after every CHUNK_SIZE of data has been sent, the sender waits until the receiver has written all the data to the file before beginning to send the next CHUNK.
***object-based data transfer
According to the authors of FOBS,
"the fundamental characteristic of an object-based data transfer that is leveraged by FOBS is the assumption that the user-level data buffer spans the entire object to be transferred.... this characteristic allows FOBS to push to the limit the basic concept of the "Large Window" extensions developed for TCP."
***user interface
There is a java drag-n-drop interface that has been written for FOBS however I had trouble getting it to always complete a file transfer. The command line version seemed to be more reliable.
***rate control
The sending rates are calculated before the transfer begins and adjusted using the pre-calculated rates pulled from an array as the flow continues. Also the rate control mechanism checks the actual sending rate after a specified number of packets(25) have been sent--not after every packet.