x86-64 TUTORIAL: CALLING CONVENTIONS

SYSTEM CALLS

System calls are made using the syscall instruction on an x86-64 version of GNU/Linux as opposed to using int 0x80 on an x86 version of GNU/Linux. All programs are in long mode. Depending on the type of GNU/Linux system you use, the list of system calls can be found in /usr/include/asm/unistd_64.h for Debian-based systems or in /usr/include/asm-x86_64/unistd.h for Slackware, etc.

CALLING CONVENTION

System Calls

• The kernel or system call interface uses registers RDI, RSI, RDX, R10, R8, R9, respectively, for passing arguments in that order. A maximum of 6 parameters can be passed.
• The kernel destroys registers RCX and R11.
• The number of the system call is passed in the register RAX.
• No argument is passed on the stack.
• The return value is placed in RAX. An error value is in the range -1 to -4095 (0xFFFFFFFF to 0xFFFF0000).
• In 32-bit mode, GNU/Linux supports 6 arguments in the system call and they are passed in the registers EBX, ECX, EDX, ESI, EDI and EBP, with the system call number in EAX.

Function Calls

• For a complete list of the registers that should be used for passing parameters and for return values, refer the x86-64 ABI.
• The integer and pointer arguments are passed in the registers RDI, RSI, RDX, RCX, R8, R9 in that order.
• The registers XMM0-XMM7 are used to pass the single and double precision floating point arguments.
• Rest of the arguments might have to be passed on the stack.
• The RAX register should hold the number of SSE registers (XMM0-XMM7) that are used in the passing of arguments.
• The registers RBX, RSP, RBP, R12-R15 are callee-saved registers and are preserved across function calls.
• RBX is the optional base pointer and RBP is the optional frame pointer.
• R11 is the temporary register used by the Procedure Linkage Table.
• R10 is used to pass a function’s static chain pointer.
• Integer or pointer type return values are returned in RAX and RDX.
• Floating point return values are returned in XMM0 and XMM1.
• Long double precision values are returned in ST(0) and ST(1).