Probing for sandbox presence

A novel detection method for Linux machines

2026-04-20 * Wintrmvte

Intro

Hello there - it's been a while since my last post, and now thanks to some spare time I have a great opportunity to publish more often.

This post briefly outlines a new, reliable solution for checking if an implant was run inside an isolated/virtualized environment. The logic is written in 64-bit Assembly (NASM) and ensures that output is a fully position independent shellcode.

Motivation

Most existing methods either use differential timing analysis with cpuid instruction (which itself is heavily flagged by security products) or are based on horizontal reconaissance of the system resources. Also, authors often embedd some kind of hyperisor-intensive loop in order to obtain timing baseline/speed of execution. I will therefore try to avoid both the often abused instruction itself, as well any complex for/while sequences in my code.

Approach

The quirk itself is based on how Linux kernel handles the Speculative Execution Side Channel operations, and how their timing differs in a baremetal vs. sandbox execution context. On a real Linux machine with a modern CPU, this is a fast cache-level toggle. In a sandbox, the hypervisor must first verify if the guest is allowed to control speculation mitigations, triggering a heavy and resource intensive VM-exit, and offloading the task to the physical resources.

For example, on QEMU/KVM setup, operations related to speculative branching are intercepted and slowed down, in order to prevent L1 Cache Terminal Fault. This timing inconsistency allows us to determine if we are trapped inside a VM by using only two consecutive system calls. I chose the Speculative Control Toggle provided by the prtcl interface in order to analyze time difference between a standard system call and one related to CPU speculation.

Part I - setup

First comes a definition of a macro for aligning the cycles result in r12, as well as some system call numbers and variables. Then, we can obtain the value of the initial timing:

[bits 64]
 
%define SYS_GETPID 39
%define SYS_PRCTL 157
%define SYS_EXIT 60
 
%define SHIFT 32
%define SPECULATION 53
%define BYPASS 1
%define DISABLE 4
%define EINVAL -22
%define RATIO 20
 
; Store the lower half of RDX (CPU cycles) inside RAX
%macro rdt
    rdtsc
    shl rdx, SHIFT
    or rax, rdx
%endmacro
 
section .text
global _start
_start:
    rdt
    mov r12, rax

The start time is placed in register r12 - now I must calculate how many cycles does it take for a standard system call to run (getpid(2) was chosen for this example):

    push SYS_GETPID                 
    pop rax
    syscall
    rdt
    sub rax, r12
    mov r13, rax           

I stored the time it took to perform the operation in register r13.

Part II - speculative probing

Now I will invoke the speculative operation - a reconfiguration of lookahead branching state, which will cause a massive overhead if performed in a virtual machine.

The objective is to launch the following system call:

prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0x00, 0x00)

in similar way as the previous one, which was used to obtain the baseline timing. This means that I will use the rdtsc instruction again, right before and after the call to prctl() and store the result (again in r12) for further analysis:

    rdt
    mov r12, rax            ; R12 = Start Time
    mov rax, SYS_PRCTL      
    mov rdi, SPECULATION    ; PR_SET_SPECULATION_CTRL (53)
    xor rsi, rsi
    inc rsi                 ; PR_SPEC_STORE_BYPASS (1)
    push DISABLE            
    pop rdx                 ; PR_SPEC_DISABLE (4)
    xor r10, r10            ; 4th arg (null)
    xor r8, r8              ; 5th arg (null)
    syscall
 
    cmp rax, EINVAL
    je virtual
 
    rdt
    sub rax, r12

I also check the return value for -EINVAL as this might indicate a blocked/unsupported syscall, so therefore most likely a sandbox.

Part III - ratio analysis

On physical hardware, the speculative configuration change is quick. In a VM, reconfiguring hardware speculation features often requires a synchronization of all vCPUs by the Host. Setting the ratio to >= 20x baseline is a high-confidence VM indicator.

Based on above, mow it is time for the final check: after the imul instruction, which is used to calculate the ratio, I issue the cmp rax, r13 to perform the comparison - if the result exceeds the treshold, it is highly probable that we landed inside a VM.

    imul r13, r13, RATIO
    cmp rax, r13
    ja .virtual
physical:
    ; Insert continuation of the implant logic on baremetal...
    ; . . .
virtual:
    ; ... or perform example exit(-1) when virtualization is present
    xor rdi, rdi
    dec rdi
    push SYS_EXIT
    pop rax
    syscall

Code

The full source is available below:

; [spex.asm]
; VM detection based on speculative branching
; @ Wintrmvte
 
; nasm -f elf64 -o spex.o spex.asm
; ld -m elf_x86_64 -o spex spex.o 2>&1
; ./spex
 
[bits 64]
 
%define SYS_GETPID 39
%define SYS_PRCTL 157
%define SYS_EXIT 60
 
%define SHIFT 32
%define SPECULATION 53
%define BYPASS 1
%define DISABLE 4
%define EINVAL -22
%define RATIO 20
 
%macro rdt
    rdtsc
    shl rdx, SHIFT
    or rax, rdx
%endmacro
 
 
section .text
global _start
_start:
    rdt
    mov r12, rax
    push SYS_GETPID                 
    pop rax
    syscall
    rdt
    sub rax, r12
    mov r13, rax           
    rdt
    mov r12, rax            
    mov rax, SYS_PRCTL      
    mov rdi, SPECULATION    
    xor rsi, rsi
    inc rsi                 
    push 4
    pop rdx                 
    xor r10, r10            
    xor r8, r8              
    syscall
    cmp rax, EINVAL
    je virtual
    rdt
    sub rax, r12
    imul r13, r13, RATIO
    cmp rax, r13
    ja .virtual
physical:
    ; <you-code-here>
virtual:
    xor rdi, rdi
    dec rdi
    push SYS_EXIT
    pop rax
    syscall