Calling User specified PROTOCOL from Python

Suppose:
There is PORTOCOL which is not defined in UEFI/PI and is built in BIOS image.
This PROTOCOL interface is known by C language header file:

#ifndef _XYZ_DXE_H_
#define _XYZ_DXE_H_

#define USER_XYZ_PROTOCOL_GIUD \
  { \
    0x2a339172, 0x7ef6, 0x4f2b, { 0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7} \
  }


typedef struct _USER_XYZ_PROTOCOL USER_XYZ_PROTOCOL;

typedef
EFI_STATUS
(EFIAPI *USER_XYZ_GET_VERSION)(
  IN  CONST USER_XYZ_PROTOCOL     *This,
  OUT UINT64                      *VersionSize
  );

typedef
EFI_STATUS
(EFIAPI *USER_XYZ_DUMMY)(
  VOID
  );

///
/// XYZ Protocol structure
///
struct _USER_XYZ_PROTOCOL {
  USER_XYZ_GET_VERSION    GetVersion;
  USER_XYZ_DUMMY          Dummy;
};

#endif // _XYZ_DXE_H_

PROTOCOL's EfiPy2 form and How to use it:
#
# xyzProtocol.py
#

#
# XYX PROTOCOL declare in EfiPy2 format
#
from EfiPy2 import *

#
# XYZ protocol GUID
#
# #define USER_XYZ_PROTOCOL_GIUD \
#   { \
#     0x2a339172, 0x7ef6, 0x4f2b, { 0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7} \
#   }
# EFI_GUID gUserXyzProtocolGuid = USER_XYZ_PROTOCOL_GIUD;
#
gUserXyzProtocolGuid                 = \
  EFI_GUID (0x2a339172, 0x7ef6, 0x4f2b, (0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7))

#
# XYZ protocol declare
#
# typedef struct _USER_XYZ_PROTOCOL USER_XYZ_PROTOCOL;
#
class USER_XYZ_PROTOCOL (Structure):
  pass

#
# USER_XYZ_GET_VERSION function declare
#
# typedef
# EFI_STATUS
# (EFIAPI *USER_XYZ_GET_VERSION)(
#   IN  CONST USER_XYZ_PROTOCOL     *This,
#   OUT UINT64                      *VersionSize
#   );
#
USER_XYZ_GET_VERSION = CFUNCTYPE (
  EFI_STATUS,
  POINTER (USER_XYZ_PROTOCOL),
  POINTER (UINT64)
  )

#
# USER_XYZ_DUMMY function declare
#
# typedef
# EFI_STATUS
#   (EFIAPI *USER_XYZ_DUMMY)(
#   VOID
#   );
#
# Return type: EFI_STATUS
# Input parameter: None
#
USER_XYZ_DUMMY = CFUNCTYPE (
  EFI_STATUS
  )

#
# XYZ Protocol structure
#
# struct _USER_XYZ_PROTOCOL {
#   USER_XYZ_GET_VERSION    GetVersion;
#   USER_XYZ_DUMMY          Dummy;
# };
#
USER_XYZ_PROTOCOL._fields_ = [
  ("GetVersion",  USER_XYZ_GET_VERSION),
  ("Dummy",       USER_XYZ_DUMMY)
  ]


#
# Test program for verifying protocol conversation
#
if __name__ == '__main__':

  Interface = PVOID ()
  Status = gBS.LocateProtocol (
                 byref (gUserXyzProtocolGuid),
                 None,
                 byref (Interface)
                 )

  if Status != 0x00:
    print "Locate Protocol Error (Status:%x)" % Status
    exit (0)

  XyzProtocol = cast (Interface, POINTER(USER_XYZ_PROTOCOL))

  Version = UINT64(0)

  Status = XyzProtocol[0].GetVersion (XyzProtocol, byref(Version))

  print "Get Version: %x (Status:%x)" % (Version.value, Status)

Advanced
After this EfiPy2 protocol program test is done, it can be put into package folder, including __init__.py.

Setting PYTHONPATH uefi shell environment variable lets user specified protocol in anywhere.

Example: Folder fs0:\Xyz includes xyzProtocol.py and __init__.py 

set PYTHONPATH fs0:\Xyz

USER_XYZ_PROTOCOL can be called from anywhere by

import xyzProtocol

Q&A: Should EfiPy2 programmer needs to reference protocol implementation source code?
Answer: 
No, it is the same as EFI caller program, EfiPy2 programmer needs Protocol header file and transfer it to Python code, only.

Q&A: Should python.efi needs to be compiled, again, for user specified PROTOCOL?
Answer:
No, python.efi includes ctypes.
ctypes for UEFI is a foreign function library for Python. It provides C compatible data types, and allows calling functions in UEFI PROTOCOL. It can be used to wrap PROTOCOLs in pure Python. (ref. from Here)

Python package in UEFI shell environment

Intrdocution

It is short description for Python package file structure.

Assume following folder structure in boot device

FSx: Efi

     ├── Apps

     │   ├── CorepySample

     │   └── EfiPy2Sample

     ├── BOOT

     │   └── BOOTX64.EFI

     ├── lib

     ├── StdLib

     │   ├── etc

     │   └── lib

     │       └── python36.8

     │           ├── corepy

     │           ├── ctypes

     │           ├── EfiPy2

     │           │   ├── Lib

     │           │   ├── MdePkg

     │           │   │   ├── Base.py

     │           │   │   ├── Guid

     │           │   │   ├── IndustryStandard

     │           │   │   ├── Library

     │           │   │   ├── Pi

     │           │   │   ├── Ppi

     │           │   │   ├── Protocol

     │           │   │   ├── Register

     │           │   │   │   ├── Amd

     │           │   │   │   └── Intel

     │           │   │   │       ├── Microcode.py

     │           │   │   │       └── Msr

     │           │   │   └── Uefi

     │           │   └── ShellPkg

     │           │       ├── Guid

     │           │       └── Protocol

     │           └ ... <<Python linraries>>

     └── Tools

         └── Python.efi

Python.efi is at FSx:\EFI\Tools and key folders are...

1. EfiPy2 self-defined modules are at FSx:\Efi\StdLib\lib\python36.8\EfiPy2\Lib

2. Executable python tool is at FSx:\EFI\Tools.

Any python scripts which using EfiPy2 packages in FS0:\EFI\Apps need to have import statement.

import EfiPy2

or

import EfiPy2 as EfiPy

Example

This script file "Startup.nsh" run "CpuIdBasic.py", automatically, while system boot into shell.

(Assume python and its related packages are installed in fs0)

Python.efi fs0:\Efi\Apps\EfiPy2Sample\CpuIdBasic.py

dump SMBIOS data with EfiPy

Sample code:
https://github.com/EfiPy/EfiPy2/blob/main/Efi/Apps/EfiPy2Sample/EfiPy2Smbios.py

Description:
This sample uses EFI_SMBIOS_PROTOCOL, defined from EfiPy2.MdePkg.Protocol.Smbios.

By using EFI_SMBIOS_PROTOCOL protocol member function GetNext(), it has the same algorithm as EDK II driver/application. infinite while loop until return error in GetNext().


In this sample, it uses getattr() member function to get SMBIOS type class defined in EfiPy2.MdePkg.IndustryStandard.SmBios.

This sample also leverages the power of Python/ctypes (sample in the function SmbiosDumpStruct2() and SmbiosDumpStruct3()).

In this function, it does not need to know SMBIOS structure definition, it uses Python getattr() to get SMBIOS structure member name and its value.

gRT GetTime/SetTime

Sample code name:
https://github.com/EfiPy/EfiPy2/blob/main/Efi/Apps/EfiPy2Sample/EfiPyTime.py

Description:
It is simple example of comparison between C language with EfiPy2 (ctypes)

# EFI_TIME TimeCur;
TimeCur = EfiPy2.EFI_TIME ()
# EFI_TIME_CAPABILITIES TimeCap;
TimeCap = EfiPy2.EFI_TIME_CAPABILITIES ()

# Status = gRT->GetTime (&TimeCur, &TimeCap);
Status  = EfiPy2.gRT.GetTime (EfiPy2.byref(TimeCur), EfiPy2.byref(TimeCap))

# TimeCur.Year = 2014;
TimeCur.Year = 2014
# TimeCur.Hour = 3;
TimeCur.Hour = 3

# Status = gRT->SetTime (&TimeCur);
Status  = EfiPy2.gRT.SetTime (EfiPy2.byref(TimeCur))

# Status = gRT->GetTime (&TimeCur, NULL);
Status  = EfiPy2.gRT.GetTime (EfiPy2.byref(TimeCur), None)

EfiPy2.byref is the sample as address of (&) in C language.

x86 instruction CPUID in CorePy/EfiPy

Sample code
https://github.com/EfiPy/EfiPy2/blob/main/Efi/Apps/EfiPy2Sample/CpuIdBasic.py

Description

In this demo, the sample script shows how to uses x86 instruction CPUID with CorePy/EfiPy2.
We have put CpuId module in EfiPy2.Lib in EfiPy2, that is the re-usable CpuId module need to be import as

from EfiPy2.Lib import CpuId

In CpuId module, it has class named "CpuIdClass". and create object as

CpuIdObj  = CpuId.CpuIdClass()

CpuId module also provide generic register suit for CPUID instruction operation by CorePy.

CpuIdReg  = CpuId.CPUID_GENERIC_REGISTERs()

CpuIdClass has two member functions... GetId () and GetId2 (). Both functions take three parameters. The first two parameters are EAX and ECX by instruction CPUID. But the third parameter is memory address by GetId (),  CPUID_GENERIC_REGISTERs object for GetId2 ().

Also, user can create its own register with pack structure type for CPUID, for example...

import EfiPy2.MdePkg.Register.Intel.Cpuid as CpuidRegs

class CpuIdRegisters (EfiPy.Structure):

  _pack_   = 1

  _fields_ = [

    ('EAX',  CpuidRegs.CPUID_VERSION_INFO_EAX),

    ('EBX',  CpuidRegs.CPUID_VERSION_INFO_EBX),

    ('ECX',  CpuidRegs.CPUID_VERSION_INFO_ECX),

    ('EDX',  CpuidRegs.CPUID_VERSION_INFO_EDX)

  ]

CpuIdReg  = CpuIdRegisters()

This registers can be used to get CPU version infor by...

rax = CpuIdObj.GetId2 (CpuidRegs.CPUID_VERSION_INFO, 0, CpuIdReg)

Due to CpuidRegs.CPUID_VERSION_INFO_EAX, CpuidRegs.CPUID_VERSION_INFO_EBX, CpuidRegs.CPUID_VERSION_INFO_ECX and CpuidRegs.CPUID_VERSION_INFO_EDX are union type including each bit field, Python script can also dump version information detail.

PCI Scan

Sample Code Name
https://github.com/EfiPy/EfiPy2/blob/main/Efi/Apps/EfiPy2Sample/PciScan.py

Description

PCI Scanning program is the sample of how data returned by CorePy then transfer to EfiPy2.

It uses below functions, constants and structure in  EfiPy2

EfiPy2.MdePkg.IndustryStandard.Pci
Structure PCI_TYPE_GENERIC
Structure PCI_CONFIG_ACCESS_CF8
PCI_MAX_BUS
PCI_MAX_DEVICE
PCI_MAX_FUNC
IS_PCI_MULTI_FUNC

CorePy code in PCI scanning script, it uses I/O port 0x0cf8, 0x0cfc for scanning PCI generic register.

It also needs input parameter for assigning PCI bus, Device, function and registers.

EfiPy2 full package is from https://github.com/EfiPy/EfiPy2

Pseudo Sample Code

class PciScan:

  #
  # Prebuild assembly code
  #
  def __init__ (self):
    self.code.add(x86.mov(reg.rax, mem.MemRef(reg.rbp, 16)))
    self.code.add(x86.mov(reg.dx, 0x0cf8))
    self.code.add(x86.out(reg.dx, reg.eax))

    self.code.add(x86.mov(reg.dx, 0x0cfc))
    self.code.add(x86.in_(reg.eax, reg.dx))

  #
  # Get PCI register entry, return UINT32 value
  #
  def scan (self, Bus = 0, Dev = 0, Func = 0, Reg = 0):
    reg = pci.PCI_CONFIG_ACCESS_CF8((Reg & 0xFC, Func, Dev, Bus, 0, 1))

    self.params.p1 = reg.Uint32
    ret = self.proc.execute(self.code,
                            params = self.params,
                            mode = 'int')

    return ret

if __name__ == '__main__':

  PciDev  = PciScan()
  ret = PciDev.scan(Bus, Dev, Func, Reg)

Result

CorePy (Python Assembly language on EFI Shell)

Sample code name
https://github.com/EfiPy/EfiPy2/blob/main/Efi/Apps/CorepySample/asm.py

Description
For low level controlling by Python language in UEFI shell environment, EfiPy2 leverages another open source project CorePy.

Sample Code: output 0xAA to port 0x80
1. Download latest EfiPy from https://github.com/EfiPy/EfiPy2

2. Copy Disk/EFI folder to boot device, which does not includes bootable shell, yet.
3. Copy asm.py to EFI/Tools.
4. Boot into shell and change working folder to fsx:\EFI\tools
5. Run "Python.efi asm.py"

Source code
code.add(x86.mov(dx, 0x80))
code.add(x86.mov(ax, 0xaa))
code.add(x86.out(dx, ax))

proc.execute(code)

Result

PanelTest.py and its demo video on YouTube

Introduction
This is simple python program which demos how PanelTest.py run in UEFI shell, and how the program result is after python script is edited immediately in shell.

Source code
PanelTest.py. Also, each test item in PanelTest.py is split out... PanelFont.py, PanelGrid.py and PanelRuler.py.