Tzi-cker Chiueh: Checking Array Bound Violation Using Segmentation Hardware

Black Hat Briefings, Las Vegas 2005 [Audio] Presentations from the security conference

The ability to check memory references against their associated array/buffer bounds helps programmers to detect programming errors involving address overruns early on and thus avoid many difficult bugs down the line. Because such programming errors have been the targets of remote attacks, i.e., buffer overflow attack, prevention of array bound violation is essential for the security and robustness of application programs that provide service on the Internet. This talk proposes a novel approach called CASH to the array bound checking problem that exploits the segmentation feature in the virtual memory hardware of the X86 architecture. The CASH approach allocates a separate segment to each static array or dynamically allocated buffer, and generates the instructions for array references in such a way that the segment limit check in X86's virtual memory protection mechanism performs the necessary array bound checking for free. In those cases that hardware bound checking is not possible, it falls back to software bound checking. As a result, CASH does not need to pay per-reference software checking overhead in most cases. However, the CASH approach incurs a fixed set-up overhead for each use of an array, which may involve multiple array references. The existence of this overhead requires compiler writers to judiciously apply the proposed technique to minimize the performance cost of array bound checking. This talk will describe the detailed design and implementation of the CASH compiler, and a comprehensive evaluation of various performance tradeoffs associated with the proposed array bound checking technique. For the set of production-grade network applications we tested, including Apache, Sendmail, Bind, etc., the latency penalty of CASH's bound checking mechanism is between 2.5% to 9.8% when compared with the baseline case that does not perform any bound checking. Dr. Tzi-cker Chiueh is a Professor in the Computer Science Department of Stony Brook University, and the Chief Scientist of Rether Networks Inc. He received his B.S. in EE from National Taiwan University, M.S. in CS from Stanford University, and Ph.D. in CS from University of California at Berkeley in 1984, 1988, and 1992, respectively. He received an NSF CAREER award in 1995, and has published over 130 technical papers in refereed conferences and journals in the areas of operating systems, networking, and computer security. He has developed several innovative security systems/products in the past several years, including SEES (Secure Mobile Code Execution Service), PAID (Program Semantics-Aware Intrusion Detection), DOFS (Display-Only File Server), and CASH.