RCE Endeavors 😅

Table of Contents:

• Introduction (1/11)
• Goals and Architecture (2/11)
• The inotify API (3/11)
• File Watcher, C++ (4/11)
• File Watcher, Bash (5/11)
• Execution Engine (6/11)
• Containerization (7/11)
• Kubernetes (8/11)
• Demo (9/11)
• Conclusion (10/11)
• System Setup (11/11)
• Source code

The previous post covered the first half of the file watcher component: the background C++ process responsible for monitoring the input file directory and starting the compilation and execution process. This next part will cover the helper Bash script that actually does the work. This script is composed of two parts: the main script, which is a general script responsible creating and cleaning up the directories that will be used, and language-specific scripts responsible for running the process of compilation and execution.

Main script

The main script is what is invoked by the background process. The path and name to this script is configured via the configuration file:

"bootstrappath": "${CODE_PATH}/share/bootstrap",
"bootstrapscriptname": "bootstrap-common.sh",
...

The background process builds the command to invoke this script (see NotifyChildProcess::buildCommand in the previous post), and captures its output. The main script is shown in its entirety below:

#!/bin/bash

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -f|--filepath)
    FILE_PATH="$2"
    shift
    shift
    ;;
    -a|--arguments)
    ARGUMENTS_PATH="$2"
    shift
    shift
    ;;
    -i|--index)
    INDEX="$2"
    shift
    shift
    ;;
    -d|--dependenciespath)
    DEPENDENCIES_PATH="$2"
    shift
    shift
    ;;
    -w|--workspacepath)
    WORKSPACE_PATH="$2"
    shift
    shift
    ;;
    -o|--outputpath)
    OUTPUT_PATH="$2"
    shift
    shift
    ;;
    -s|--stdinPath)
    STDIN_PATH="$2"
    shift
    shift
    ;;
    -t|--timeout)
    INTERACTIVE_TIMEOUT="$2"
    shift
    shift
    ;;
    -l|--language)
    LANGUAGE="$2"
    shift
    shift
    ;;
    *)
    
    POSITIONAL+=("$1")
    shift
    ;;
esac
done
set -- "${POSITIONAL[@]}"

FILE_NAME="$(basename ${FILE_PATH})"
ARGS_FILE_EXISTS=false
STDIN_FILE_EXISTS=false

OUTPUT_NAME=${WORKSPACE_PATH}/${INDEX}/${FILE_NAME}

if [ -f "${ARGUMENTS_PATH}" ]; then
    ARGS_FILE_EXISTS=true
fi

if [ -f "${STDIN_PATH}" ]; then
    STDIN_FILE_EXISTS=true
fi

create_directories () {
    rm -rf ${WORKSPACE_PATH}/${INDEX}
    mkdir ${WORKSPACE_PATH}/${INDEX}
}

cleanup () {
    rm ${FILE_PATH}
    if [ "${ARGS_FILE_EXISTS}" = "true" ]; then
        rm ${ARGUMENTS_PATH}
    fi
    if [ "${STDIN_FILE_EXISTS}" = "true" ]; then
        rm ${STDIN_PATH}
    fi
    rm -rf ${WORKSPACE_PATH}/${INDEX}
}

copy_dependencies () {
    cp ${FILE_PATH} ${WORKSPACE_PATH}/${INDEX}
    cp -r ${DEPENDENCIES_PATH}/. ${WORKSPACE_PATH}/${INDEX}
}

move_output () {
    rm -rf ${OUTPUT_PATH}/${INDEX}
    mkdir ${OUTPUT_PATH}/${INDEX}
    mv ${OUTPUT_NAME}-output.log ${OUTPUT_PATH}/${INDEX}
}

main () {
    create_directories
    copy_dependencies
    (cd ${CODE_PATH}/share/${LANGUAGE}/bootstrap/${LANGUAGE}; source ./bootstrap.sh; run_command)
    move_output
    cleanup
    
    exit ${result}
}

main

Despite the large size of the script, half of it is just related to argument parsing and storage. The other half is for the actual functionality: creating the isolated workspace directory that the compilation and execution will take place in (create_directories function), copying the dependencies to this workspace directory (copy_dependencies function), moving the output to the output directory and cleaning up (move_output and cleanup functions). The run_command function is the language-specific function that will get invoked from its respective language directory.

Language-specific scripts

Each language has its own way to go from source code to executable. Compiled languages require a compilation step to generate an executable, while interpreted ones like Python can have the interpreter run directly on the source code. As a result, each language has its own language-specific script responsible for implementing this functionality.

For example, the run_command implementation for C files looks like this:

#!/bin/bash

run_command () {

    TIMEOUT_SECONDS_COMPILE=15s
    TIMEOUT_SECONDS_RUN=10s

    timeout ${TIMEOUT_SECONDS_COMPILE} gcc -Wall -std=c17 -Wno-deprecated ${OUTPUT_NAME} -o ${OUTPUT_NAME}.out >> ${OUTPUT_NAME}-output.log 2>&1
    result=$?
    
    if [ $result -eq 0 ]
    then
        chmod 753 ${OUTPUT_NAME}-output.log
        if [ "${ARGS_FILE_EXISTS}" = "true" ]; then
            ARGUMENTS=$(cat ${ARGUMENTS_PATH})
        fi
        if [ "${STDIN_FILE_EXISTS}" = "true" ]; then
            TIMEOUT_SECONDS_RUN=${INTERACTIVE_TIMEOUT}
            STDIN_ARGUMENTS="-s ${STDIN_PATH}"
        fi
        
        sudo su-exec exec ${EXEC_PATH}/executor -t ${TIMEOUT_SECONDS_RUN} ${STDIN_ARGUMENTS} -o ${OUTPUT_NAME}-output.log -f ${OUTPUT_NAME}.out ${ARGUMENTS}
        result=$?
    fi
}

This function begins by invoking the GCC compiler to create an executable. The output of this process is captured via stream redirection into an output file, so if the compilation fails then the cause will be still be captured and presented to the user. Once the executable is generated, the execution component is called to run it. Both of these steps come with a timeout value in order to prevent issues with hanging compiler processes or infinitely running executables.

Once the execution component has completed, the process return code is returned to the main script and subsequently the background process. The implementation of the execution component will be covered in-depth in the next post. After the discussion of the execution component, the end-to-end details of how the system works will be complete.