Memory Mapping using Python in OS X

Whenever it comes to Inter Process Communication, shared memory seems to be the fastest way of doing it. Once the memory is mapped into the address space in memory region , no further kernel involvement is required. Thats awesome! Cuz, unlike other methods (FIFO,messages) you subvert system calls, which makes it blazingly fast.

We will be going through some basics of mmap (memory maping) using Python. Fortunately, python has a builtin module called mmap which exposes most of the calls using api.
I have a sample file pgm1.go which has a simple go program in it (Dont worry I am not going into go programming language). We are just mapping the contents of the file into the memory and reading its content.

import mmap
with open('pgm1.go', 'r+b') as f:
  mm = mmap.mmap(f.fileno(),0)
  for i in mm:
   print mm.readline()

Now, how do we verify the data is actually in memory. This is where the tool pmap is very helpfull. But wait a sec! , we dont have that in OS X do we ? Atleast I am not aware of it. Fortunately OS X comes with vmmap, which beautifully displays the same information.

besidecome-lm:pointers rahulram$ vmmap 71058
==== Summary for process 71058
ReadOnly portion of Libraries: Total=57.9M resident=16.4M(28%) swapped_out_or_unallocated=41.5M(72%)
Writable regions: Total=19.9M written=1936K(10%) resident=2556K(13%) swapped_out=0K(0%) unallocated=17.4M(87%)

REGION TYPE                      VIRTUAL
===========                      =======
MALLOC                             11.2M        see MALLOC ZONE table below
MALLOC guard page                    32K
MALLOC metadata                     356K
STACK GUARD                        56.0M
Stack                              8192K
__DATA                             1296K
__LINKEDIT                         47.8M
__TEXT                             10.0M
__UNICODE                           544K
mapped file                           4K
shared memory                         8K
===========                      =======
TOTAL                             135.2M

There you see the mapped file is shown to be of size 4K. In your python interpreter you can
mm.close()
and see vmmap will not list that info. Yes I trust vmmap and take it for granted :P

Lets take another example involving multiple process

import mmap
import os

with open('pgm1.go', 'r+b') as f:
      mm = mmap.mmap(f.fileno(),0)

print os.getpid()
pid = os.fork() # Creating a child process

if pid == 0: 
    mm.seek(0)
    print os.getpid()
    print os.getppid()
    print mm.readline()
    mm.close()

The child process is basically reading the content from mmap and closing the same.

Note: if the mm object is closed before other processes are using it, then it will start throwing an exception. Please ensure mm object is closed only after flushing the content to disk