A novel detection method for Linux machines
2026-04-20 * Wintrmvte
Hello there - it's been a while since my last post, and now thanks to some spare time I have a great opportunity to publish more often.
This post briefly outlines a new, reliable solution for checking if an implant was run inside an isolated/virtualized environment. The logic is written in 64-bit Assembly (NASM) and ensures that output is a fully position independent shellcode.
Most existing methods either use differential timing analysis with cpuid instruction (which itself is heavily flagged by security products) or are based on horizontal reconaissance of the system resources. Also, authors often embedd some kind of hyperisor-intensive loop in order to obtain timing baseline/speed of execution. I will therefore try to avoid both the often abused instruction itself, as well any complex for/while sequences in my code.
The quirk itself is based on how Linux kernel handles the Speculative Execution Side Channel operations, and how their timing differs in a baremetal vs. sandbox execution context. On a real Linux machine with a modern CPU, this is a fast cache-level toggle. In a sandbox, the hypervisor must first verify if the guest is allowed to control speculation mitigations, triggering a heavy and resource intensive VM-exit, and offloading the task to the physical resources.
For example, on QEMU/KVM setup, operations related to speculative branching are intercepted and slowed down, in order to prevent L1 Cache Terminal Fault. This timing inconsistency allows us to determine if we are trapped inside a VM by using only two consecutive system calls. I chose the Speculative Control Toggle provided by the prtcl interface in order to analyze time difference between a standard system call and one related to CPU speculation.
First comes a definition of a macro for aligning the cycles result in r12, as well as some system call numbers and variables. Then, we can obtain the value of the initial timing:
[bits 64]
%define SYS_GETPID 39
%define SYS_PRCTL 157
%define SYS_EXIT 60
%define SHIFT 32
%define SPECULATION 53
%define BYPASS 1
%define DISABLE 4
%define EINVAL -22
%define RATIO 20
; Store the lower half of RDX (CPU cycles) inside RAX
%macro rdt
rdtsc
shl rdx, SHIFT
or rax, rdx
%endmacro
section .text
global _start
_start:
rdt
mov r12, raxThe start time is placed in register r12 - now I must calculate how many cycles does it take for a standard system call to run (getpid(2) was chosen for this example):
push SYS_GETPID
pop rax
syscall
rdt
sub rax, r12
mov r13, rax I stored the time it took to perform the operation in register r13.
Now I will invoke the speculative operation - a reconfiguration of lookahead branching state, which will cause a massive overhead if performed in a virtual machine.
The objective is to launch the following system call:
prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0x00, 0x00)
in similar way as the previous one, which was used to obtain the baseline timing.
This means that I will use the rdtsc instruction again, right before and after the call to prctl() and store the result (again in r12) for further analysis:
rdt
mov r12, rax ; R12 = Start Time
mov rax, SYS_PRCTL
mov rdi, SPECULATION ; PR_SET_SPECULATION_CTRL (53)
xor rsi, rsi
inc rsi ; PR_SPEC_STORE_BYPASS (1)
push DISABLE
pop rdx ; PR_SPEC_DISABLE (4)
xor r10, r10 ; 4th arg (null)
xor r8, r8 ; 5th arg (null)
syscall
cmp rax, EINVAL
je virtual
rdt
sub rax, r12I also check the return value for -EINVAL as this might indicate a blocked/unsupported syscall, so therefore
most likely a sandbox.
On physical hardware, the speculative configuration change is quick. In a VM, reconfiguring hardware speculation features often requires a synchronization of all vCPUs by the Host. Setting the ratio to >= 20x baseline is a high-confidence VM indicator.
Based on above, mow it is time for the final check: after the imul instruction, which is used to calculate the ratio, I issue the
cmp rax, r13 to perform the comparison - if the result exceeds the treshold, it is highly probable that we landed inside a VM.
imul r13, r13, RATIO
cmp rax, r13
ja .virtual
physical:
; Insert continuation of the implant logic on baremetal...
; . . .
virtual:
; ... or perform example exit(-1) when virtualization is present
xor rdi, rdi
dec rdi
push SYS_EXIT
pop rax
syscallThe full source is available below:
; [spex.asm]
; VM detection based on speculative branching
; @ Wintrmvte
; nasm -f elf64 -o spex.o spex.asm
; ld -m elf_x86_64 -o spex spex.o 2>&1
; ./spex
[bits 64]
%define SYS_GETPID 39
%define SYS_PRCTL 157
%define SYS_EXIT 60
%define SHIFT 32
%define SPECULATION 53
%define BYPASS 1
%define DISABLE 4
%define EINVAL -22
%define RATIO 20
%macro rdt
rdtsc
shl rdx, SHIFT
or rax, rdx
%endmacro
section .text
global _start
_start:
rdt
mov r12, rax
push SYS_GETPID
pop rax
syscall
rdt
sub rax, r12
mov r13, rax
rdt
mov r12, rax
mov rax, SYS_PRCTL
mov rdi, SPECULATION
xor rsi, rsi
inc rsi
push 4
pop rdx
xor r10, r10
xor r8, r8
syscall
cmp rax, EINVAL
je virtual
rdt
sub rax, r12
imul r13, r13, RATIO
cmp rax, r13
ja .virtual
physical:
; <you-code-here>
virtual:
xor rdi, rdi
dec rdi
push SYS_EXIT
pop rax
syscall