The Logic of an Antivirus Software

The job of an antivirus software is to find out if an application is a virus or not. So, an antivirus software must do two things very well.

To protect itself from the virus
To correctly and quickly identify if an application is a virus

Job 1: Protecting itself from the antivirus application

In order to protect itself from an antivirus application, the antivirus software has a self-protection engine to protect its own registry settings and to protect its own installation. Also, the antivirus application has its own sandbox. So, the antivirus doesn’t run in the environment of the operating system. It in fact simulates a sandbox environment and runs in the sandbox. Any virus application trying to attack the antivirus registry or files will then be noted as the virus application may assume that the antivirus software is running in the operating system environment.

Job 2: To correctly and accurately identify a virus application

Let’s assume that you’re writing an application to identify if an application is a virus application or it is a safe application.

You will need a scanner that can scan all the files in the computer
Then, maybe the easiest place to start with is the file name. Your application should have a database of all the names of virus applications and its file sizes. So, if the file name and the file size match within the range, then you might want to flag it as a virus; as a basic anti-virus mechanism. But, obviously, this is not a fool proof way and it might flag tons of false positives.Nevertheless, it is one strategy.
So, you move to the next method. Match the hash of the file and the hash of the files from the virus database. If the hashes match, then you could be very positive that this is a virus file from the database. But, virus writers can dodge this by introducing a program to insert some random bytecodes in the object file every time so as to change the hash of the file.
So, you move to the next method which is to match the hashes of certain sections of the file, not for the entire file at once. Hash code matches and hex matches for a series of sections of the file is a fairly accurate way to determine if a file is a virus matching from the database
The other way to look at a file if it is a virus or not is by checking the processes and threads that the potential virus file is creating and destroying. The first thing to protect the user from, is the launching of malicious processes. This is the basic thing. Antivirus should register a PsSetCreateProcessNotifyRoutineEx callback. By doing this, on each process creation, and before the main thread starts to run (and cause malicious things) the antivirus callback is notified and receives all the necessary information. It receives the process name, the file object, the PID, and so. As the process is pending, the driver can tell its service to analyse the process’s memory for anything malicious. If it finds something, then the driver will simply set CreationStatus to FALSE and return.
Viruses can load image files that have malicious code. An image is a PE file, either an EXE, a DLL or SYS file. To be notified of loaded images, simply register PsSetLoadImageNotifyRoutine. That callback allows us to be notified when the image is loaded into virtual memory, even if it’s never executed. We can then detect when a process attempts to load a DLL, to load a driver, or to fire a new process.

The callback gets information about the full image path (useful for static analysis), and the more important in my opinion, the Image base address (for in-memory analysis). If the image is malicious the antivirus can use little tricks to avoid the execution, like parsing the in-memory image and go to the entrypoint, then call the assembly opcode “ret” to nullify it.

An antivirus software needs to scan a folder at wind speed.

Every read/write function made in the filesystem should go through the antivirus (AV) program.
An antivirus should be able to scan files when a user opens a folder, an archive, or when it’s downloaded on the disk. Moreover, an antivirus should be able to protect itself, by forbidding any program to delete its files.
A driver called the minifilter is introduced to able to register callbacks on every Read/Write operation made on the file system (IRP major functions). An IRP (Interrupt Request Paquet) is an object used to describe a Read/Write operation on the disk, which is transmitted along with the driver stack. The minifilter will simply be inserted into that stack, and receive that IRP to decide what to do with it (allow/deny the operation of the program).

In summary, an antivirus program has many engines or parts to it.

Engine 1: Periodically scan all executables on the disk. Pass the files through a signature engine that stores the signatures in terms of file name, size, hash, header signatures, program flow signatures, code signatures, etc.

Engine 2: Another engine is looking specifically at all files touched by users.

Engine 3: Looking at all processes running (Task manager) and checking for malicious behavior – process hollowing, code injections, writing to Windows registry, writing to sensitive files, etc.

Engine 4: Web Engine of the AntiVirus running on the browser looking for known exploits in JavaScript and web-browser

Engine 5: Network Engine: Another engine is checking all the network packets entering the network and scanning them for known exploit patterns

Engine 6: Another engine is checking all the URLs that the user is clicking and using anywhere, and checking those URLs against a database of bad URLs/IP addresses. This engine also blocks untrustworthy and inappropriate sites.

Hope this is useful to provide a brief overview of the antivirus program, thank you.