Memory Mapping using Python in OS X
Whenever it comes to Inter Process Communication, shared memory seems to be the fastest way of doing it. Once the memory is mapped into the address space in memory region , no further kernel involvement is required. Thats awesome! Cuz, unlike other methods (FIFO,messages) you subvert system calls, which makes it blazingly fast.
We will be going through some basics of mmap (memory maping) using Python. Fortunately, python has a builtin module called mmap which exposes most of the calls using api.
I have a sample file pgm1.go which has a simple go program in it (Dont worry I am not going into go programming language). We are just mapping the contents of the file into the memory and reading its content.
import mmap
with open('pgm1.go', 'r+b') as f:
mm = mmap.mmap(f.fileno(),0)
for i in mm:
print mm.readline()
Now, how do we verify the data is actually in memory. This is where the tool pmap is very helpfull. But wait a sec! , we dont have that in OS X do we ? Atleast I am not aware of it. Fortunately OS X comes with vmmap, which beautifully displays the same information.
besidecome-lm:pointers rahulram$ vmmap 71058
==== Summary for process 71058
ReadOnly portion of Libraries: Total=57.9M resident=16.4M(28%) swapped_out_or_unallocated=41.5M(72%)
Writable regions: Total=19.9M written=1936K(10%) resident=2556K(13%) swapped_out=0K(0%) unallocated=17.4M(87%)
REGION TYPE VIRTUAL
=========== =======
MALLOC 11.2M see MALLOC ZONE table below
MALLOC guard page 32K
MALLOC metadata 356K
STACK GUARD 56.0M
Stack 8192K
__DATA 1296K
__LINKEDIT 47.8M
__TEXT 10.0M
__UNICODE 544K
mapped file 4K
shared memory 8K
=========== =======
TOTAL 135.2M
There you see the mapped file is shown to be of size 4K. In your python interpreter you can
mm.close()
and see vmmap will not list that info. Yes I trust vmmap and take it for granted :P
Lets take another example involving multiple process
import mmap
import os
with open('pgm1.go', 'r+b') as f:
mm = mmap.mmap(f.fileno(),0)
print os.getpid()
pid = os.fork() # Creating a child process
if pid == 0:
mm.seek(0)
print os.getpid()
print os.getppid()
print mm.readline()
mm.close()
The child process is basically reading the content from mmap and closing the same.
Note: if the mm object is closed before other processes are using it, then it will start throwing an exception. Please ensure mm object is closed only after flushing the content to disk