Calling User specified PROTOCOL from Python

Suppose:
There is PORTOCOL which is not defined in UEFI/PI and is built in BIOS image.
This PROTOCOL interface is known by C language header file:

#ifndef _XYZ_DXE_H_
#define _XYZ_DXE_H_

#define USER_XYZ_PROTOCOL_GIUD \
  { \
    0x2a339172, 0x7ef6, 0x4f2b, { 0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7} \
  }


typedef struct _USER_XYZ_PROTOCOL USER_XYZ_PROTOCOL;

typedef
EFI_STATUS
(EFIAPI *USER_XYZ_GET_VERSION)(
  IN  CONST USER_XYZ_PROTOCOL     *This,
  OUT UINT64                      *VersionSize
  );

typedef
EFI_STATUS
(EFIAPI *USER_XYZ_DUMMY)(
  VOID
  );

///
/// XYZ Protocol structure
///
struct _USER_XYZ_PROTOCOL {
  USER_XYZ_GET_VERSION    GetVersion;
  USER_XYZ_DUMMY          Dummy;
};

#endif // _XYZ_DXE_H_

PROTOCOL's EfiPy form and How to use it:
#
# xyzProtocol.py
#

#
# XYX PROTOCOL declare in EfiPy format
#
from EfiPy import *

#
# XYZ protocol GUID
#
# #define USER_XYZ_PROTOCOL_GIUD \
#   { \
#     0x2a339172, 0x7ef6, 0x4f2b, { 0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7} \
#   }
# EFI_GUID gUserXyzProtocolGuid = USER_XYZ_PROTOCOL_GIUD;
#
gUserXyzProtocolGuid                 = \
  EFI_GUID (0x2a339172, 0x7ef6, 0x4f2b, (0x95, 0xb9, 0x83, 0xf2, 0x15, 0x18, 0xeb, 0xf7))

#
# XYZ protocol declare
#
# typedef struct _USER_XYZ_PROTOCOL USER_XYZ_PROTOCOL;
#
class USER_XYZ_PROTOCOL (Structure):
  pass

#
# USER_XYZ_GET_VERSION function declare
#
# typedef
# EFI_STATUS
# (EFIAPI *USER_XYZ_GET_VERSION)(
#   IN  CONST USER_XYZ_PROTOCOL     *This,
#   OUT UINT64                      *VersionSize
#   );
#
USER_XYZ_GET_VERSION = CFUNCTYPE (
  EFI_STATUS,
  POINTER (USER_XYZ_PROTOCOL),
  POINTER (UINT64)
  )

#
# USER_XYZ_DUMMY function declare
#
# typedef
# EFI_STATUS
#   (EFIAPI *USER_XYZ_DUMMY)(
#   VOID
#   );
#
# Return type: EFI_STATUS
# Input parameter: None
#
USER_XYZ_DUMMY = CFUNCTYPE (
  EFI_STATUS
  )

#
# XYZ Protocol structure
#
# struct _USER_XYZ_PROTOCOL {
#   USER_XYZ_GET_VERSION    GetVersion;
#   USER_XYZ_DUMMY          Dummy;
# };
#
USER_XYZ_PROTOCOL._fields_ = [
  ("GetVersion",  USER_XYZ_GET_VERSION),
  ("Dummy",       USER_XYZ_DUMMY)
  ]


#
# Test program for verifying protocol conversation
#
if __name__ == '__main__':

  Interface = PVOID ()
  Status = gBS.LocateProtocol (
                 byref (gUserXyzProtocolGuid),
                 None,
                 byref (Interface)
                 )

  if Status != 0x00:
    print "Locate Protocol Error (Status:%x)" % Status
    exit (0)

  XyzProtocol = cast (Interface, POINTER(USER_XYZ_PROTOCOL))

  Version = UINT64(0)

  Status = XyzProtocol[0].GetVersion (XyzProtocol, byref(Version))

  print "Get Version: %x (Status:%x)" % (Version.value, Status)

Advanced
After this EfiPy protocol program test is done, it can be put into package folder, including __init__.py.

Setting PYTHONPATH uefi shell environment variable lets user specified protocol in anywhere.

Example: Folder fs0:\Xyz includes xyzProtocol.py and __init__.py 

set PYTHONPATH fs0:\Xyz

USER_XYZ_PROTOCOL can be called from anywhere by

import xyzProtocol

Q&A: Should EfiPy programmer needs to reference protocol implementation source code?
Answer: 
No, it is the same as EFI caller program, EfiPy programmer needs Protocol header file and transfer it to Python code, only.

Q&A: Should python.efi needs to be compiled, again, for user specified PROTOCOL?
Answer:
No, python.efi includes ctypes.
ctypes for UEFI is a foreign function library for Python. It provides C compatible data types, and allows calling functions in UEFI PROTOCOL. It can be used to wrap PROTOCOLs in pure Python. (ref. from Here)

Build Python and EfiPy source code

Host:

    Required compiler: VS2012/VS2013
    Download EfiPy source code from https://sourceforge.net/projects/efipy/
      Assume it is untared at M:\EfiPy\
    Download UDK2015 rev #20289 from https://svn.code.sf.net/p/edk2/code/branches/UDK2015
      Assume it is downloaded to M:\EDK2
    Merge EfiPy code into UDK2015
      xcopy /E /I /Y M:\EfiPy_r0.1.1(20289)\Src M:\EDK2
    Build M:\EDK2\AppPkg\AppPkg.dsc, configurations:
      ACTIVE_PLATFORM       = AppPkg/AppPkg.dsc
      TARGET                = RELEASE
      TARGET_ARCH           = X64
      # It depends on host environment
      #TOOL_CHAIN_TAG        = VS2012x86
      # It depends on host environment
      TOOL_CHAIN_TAG        = VS2013x86
    Modify M:\EDK2\AppPkg\AppPkg.dsc

    [LibraryClasses.Common.UEFI_APPLICATION]
      EfiPy|AppPkg/Applications/Python/EfiPy.inf

    [Components]
      AppPkg/Applications/Python/PythonCore.inf{
        <LibraryClasses>
          ShellCEntryLib|AppPkg/Applications/Python/UefiShellCEntryLib/UefiShellCEntryLib.inf
      }

      For the detail how to build EDK2 code, please reference it from EDK2 official website.

Setup Python environment in disk (N:):

    mkdir N:\EFI\Boot
    mkdir N:\EFI\Tools
    copy M:\EDK2\Build\AppPkg\RELEASE_VS2013x86\X64\AppPkg\Applications\Python\PythonCore\DEBUG\Python.efi N:\EFI\Tools
    copy M:\EfiPy_r0.1.1(20289)\Src\AppPkg\Applications\Python\EfiPy\Tools\EfiPyShell.py N:\EFI\Tools
    xcopy /E /I /Y M:\Edk2\AppPkg\Applications\Python\Python-2.7.2\Lib N:\EFI\StdLib\Lib\python.27
    xcopy /E /I /Y M:\EDK2\AppPkg\Applications\Python\PyMod-2.7.2\Lib N:\EFI\StdLib\Lib\python.27
    xcopy /E /I /Y M:\EfiPy_r0.1.1(20289)\Src\AppPkg\Applications\Python\EfiPy\lib\EfiPy N:\EFI\StdLib\Lib\python.27\EfiPy

Target:

    Compile Pytghon .py file to .pyc

      Boot to EFI shell and changing working directory to fsX:\EFI\Tools

        FSX:\EFI\Tools\>Python.efi
        >>> import compileall
        >>> compileall.compile_dir('FS0:\EFI\StdLib\Lib\python.27', force=True)

SMBIOS - 2

In https://sourceforge.net/u/efipy/svn/HEAD/tree/Trunk/smbios.py r14, it adds 2 functions.

1.
Smbios.py accepts command line input parameter, which is requested SMBIOS type in numeric list.
example "Python.efi smbios.py 1 3 5", output SMBIOS information, type 1, 3, 5.

Python receives command line input parameter by Python list sys.argv.
sys.argv[0] is the program name.

2.
In StructDump() function, it output SMBIOS string if it parse SMBIOS raw data and get SMBIOS_TABLE_STRING data type.

Pseudo code of checking SMBIOS data type
#
# EfiPy/MdePkg/IndustryStandard/SmBios.py
#
class SMBIOS_TABLE_STRING (UINT8):
  pass

#
# SMBIOS sample code
#
if issubclass (field[1], Isb.SMBIOS_TABLE_STRING):
elif issubclass (field[1], EfiPy._SimpleCData):
elif isinstance (_o, EfiPy.GUID):
elif isinstance (_o, EfiPy.Structure):
elif isinstance (_o, EfiPy.Array):

Python package in UEFI shell environment

It is simple description for how to build self-defined Python package.

Assume following folder structure in boot device

FS0:--+--EFI--+--Boot
      |       |
      |       +--StdLib--lib--python.27
      |       |
      |       +--Tools
      |
      +--Tools-+-Python.efi
      |        |
      |        +-CpuIds.py
      |
      +--ToolsPkg-+-__init__.py
                  |
                  +-CpuId.py

Python.efi is at FS0:\EFI\Tools
Self-defined package is FS0:\ToolsPkg and executable python tool is at FS0:\Tools.
Any python program in FS0:\Tools will import Python module in FS0:\ToolsPkg.

First, We have to set EFI shell environment, named "PYTHONPATH".
Second, Create __init__.py in FS0:\ToolsPkg, each sub-folder has to have __init__.py, too.
Third, make sure EFI shell environment "path" including FS0:\Tools

For example, This script file "Startup.nsh" run "CpuIds.py", automatically, while system boot into shell.

fs0:
set PYTHONPATH fs0:\ToolsPkg
Python.efi fs0:\Tools\CpuIds.py

Explanation: 
fs0:\ToolsPkg folder includes __init__.py and CpuId.py, which is used by CpuIds.py in FS0:\Tools

#
# CpuIds.py
#
import CpuId.py

dump SMBIOS data with EfiPy

Sample code:
https://sourceforge.net/u/efipy/svn/HEAD/tree/Trunk/smbios.py

Description:
This sample uses EFI_SMBIOS_PROTOCOL, defined from EfiPy.MdePkg.Protocol.Smbios.

It requires EfiPy_r0.1.1(20289) due to
1. This version fixed error for SMBIOS Industry standard in EfiPy.MdePkg.IndustryStandard.SmBios
2. This version also changed data structure SMBIOS_TABLE_STRING to class.

Using EFI_SMBIOS_PROTOCOL protocol member function GetNext(). it has the same algorithm as EDK II driver/application. infinite while loop until return error in GetNext().


In this sample, it uses getattr() function to get SMBIOS type class defined in EfiPy.MdePkg.IndustryStandard.SmBios.

This sample also leverages the power of Python/ctypes (sample in the function StructDump()).
In this function, it does not need to know SMBIOS structure defined, it uses Python getattr() to get SMBIOS structure member name and its value.

gRT GetTime/SetTime

Sample code name:
https://sourceforge.net/u/efipy/svn/HEAD/tree/Trunk/Time.py

Description:
It is simple example comparison between C language and EfiPy (ctypes)

# EFI_TIME TimeCur;
TimeCur = EfiPy.EFI_TIME ()
# EFI_TIME_CAPABILITIES TimeCap;
TimeCap = EfiPy.EFI_TIME_CAPABILITIES ()

# Status = gRT->GetTime (&TimeCur, &TimeCap);
Status  = EfiPy.gRT.GetTime (EfiPy.byref(TimeCur), EfiPy.byref(TimeCap))

# TimeCur.Year = 2014;
TimeCur.Year = 2014
# TimeCur.Hour = 3;
TimeCur.Hour = 3

# Status = gRT->SetTime (&TimeCur);
Status  = gRT.SetTime (EfiPy.byref(TimeCur))
# Status = gRT->GetTime (&TimeCur, NULL);
Status  = gRT.GetTime (EfiPy.byref(TimeCur), None)

EfiPy.byref is the sample as address of (&) in C language.

x86 instruction CPUID in CorePy/EfiPy

Sample code
https://svn.code.sf.net/u/efipy/svn/Trunk/CPUID


Description

It is the demo of how to uses x86 instruction CPUID in CorePy/EfiPy.
In CPUID.py, it has python class named "CpuId".
Here is the mapping between EfiPy and C language

#
# typedef struct {
#   UINT32 EAX;
#   UINT32 EBX;
#   UINT32 ECX;
#   UINT32 EDX;
# } CpuIdStructure
#
class CpuIdStructure (EfiPy.Structure):

  _pack_   = 1
  _fields_ = [
    ('EAX',  EfiPy.UINT32),
    ('EBX',  EfiPy.UINT32),
    ('ECX',  EfiPy.UINT32),
    ('EDX',  EfiPy.UINT32)
  ]

#
# typedef union {
#   CpuIdStructure CpuInfo;
# } CpuIdUnion
#
class CpuIdUnion (EfiPy.Union):

  _pack_   = 1
  _fields_ = [
    ('CpuInfo',  CpuIdStructure)
  ]
  #
  # UINT32 GetId (UINT32 eax, UINT32 ecx, CpuIdUnion *CpuInfoAddr);
  #
  def GetId (self, eax, ecx, CpuInfoAddr):

In CpuIds.py, it is the smallest CPUID sample.

EfiPy build environment

For re-building EfiPy from sourceforge, it uses EDKII official code.
https://svn.code.sf.net/p/edk2/code/branches/UDK2015 rev: 20289

It means it get something wrong in Python package if following happen in standard output

  File "/Efi/StdLib/lib/python.27/site.py", line 69, in <module>
ImportError: No module named os

PCI Scan

Sample Code Name
https://sourceforge.net/u/efipy/svn/HEAD/tree/Trunk/PciScan.py

Description

PCI Scan program is the sample for combination between EfiPy and CorePy.

PCI scan with EfiPy, it uses below functions, constants and structure in EfiPy.MdePkg.IndustryStandard.Pci
Structure PCI_TYPE_GENERIC
Structure PCI_CONFIG_ACCESS_CF8
PCI_MAX_BUS
PCI_MAX_DEVICE
PCI_MAX_FUNC
IS_PCI_MULTI_FUNC

PCI scan with CorePy, it uses I/O port 0x0cf8, 0x0cfc for PCI generic register scanning by assembly language.
It also needs input parameter for assigning PCI bus, Device, function and registers.


EfiPy full package is from https://sourceforge.net/projects/efipy/

Pseudo Sample Code

class PciScan:

  #
  # Prebuild assembly code
  #
  def __init__ (self):
    self.code.add(x86.mov(reg.rax, mem.MemRef(reg.rbp, 16)))
    self.code.add(x86.mov(reg.dx, 0x0cf8))
    self.code.add(x86.out(reg.dx, reg.eax))

    self.code.add(x86.mov(reg.dx, 0x0cfc))
    self.code.add(x86.in_(reg.eax, reg.dx))

  #
  # Get PCI register entry, return UINT32 value
  #
  def scan (self, Bus = 0, Dev = 0, Func = 0, Reg = 0):
    reg = pci.PCI_CONFIG_ACCESS_CF8((Reg & 0xFC, Func, Dev, Bus, 0, 1))

    self.params.p1 = reg.Uint32
    ret = self.proc.execute(self.code,
                            params = self.params,
                            mode = 'int')

    return ret

if __name__ == '__main__':

  PciDev  = PciScan()
  ret = PciDev.scan(Bus, Dev, Func, Reg)

CorePy (Python Assembly language on EFI Shell)

Sample code name
https://svn.code.sf.net/u/efipy/svn/Trunk/MemIo/asm.py

Description
For low level controlling by Python language in UEFI shell environment, EfiPy leverages another open source project CorePy.

Sample Code: output 0xAA to port 0x80
1. Download latest EfiPy from https://sourceforge.net/projects/efipy/
2. Copy Disk/EFI folder to boot device, which does not includes bootable shell, yet.
3. Copy asm.py to EFI/Tools.
4. Boot into shell and change working folder to fsx:\EFI\tools
5. Run "Python.efi asm.py"

code.add(x86.mov(dx, 0x80))
code.add(x86.mov(ax, 0xaa))
code.add(x86.out(dx, ax))

proc.execute(code)

PanelTest.py and its demo video on YouTube

[Introduction]
This is simple python program, which demos how to run PanelTest.py in UEFI shell and how to edit PanelTest.py in shell environment shell.

[Note]
newest release "EfiPy_r0.0.5(1756_0005).tgz" in https://sourceforge.net/projects/efipy/ includes this demo program.