Literate programming – effect on performance

After my introductory post on Literate Programming, it occurred to me that while the concept of being able to create documentation that includes variables from the code being run is amazing, this will obviously have some impact on performance.  At best, this would be the resource required to compile the document as if it was static, while the “at worst” scenario is conceptually unbounded.  Somewhere along the way, pweave is adding extra code to pass the variables back and forth between the python and the , how and when it does this could have implications that you wouldn’t see in a simple example but could be catastrophic when running the kind of neural nets that my department are putting together. So, being a scientist, I decided to run a few experiments….¹

I needed some code that would stretch the resources of my machines and run reliably as native python and then adapt with ever increasing complexity of code.  The experiment was simple enough as I sought to determine:

1. RAM usage
2. CPU impact
3. GPU impact²
4. Time to train models

As a training exercise I took the MNist Chainer example.  This is cut and pastable code and, as long as you’ve installed the dependencies correctly, will just work.

For the side I created multiple versions of the code with the following variances:

• No markup – code to run natively as python, actually created using the pweave tangle command to ensure that I was comparing the actually python running in pweave rather than the pre-modified version (see below)
• Minimal – document definition, title and output of code only.  This would test the Pweave overhead itself and would be a useful benchmark where we’d need to output frequently changing parameters.
• Moderate markup – title, a few paragraphs of text, table and graph of results.  This would be the most general use case for internal result reporting.
• Major markup – sty file, imported images, multiple generated tables and graphs.  This would be for automatic generation of external facing documents.

Part of this creation required some adjustment of the MNist code.  Pweave is sensitive to two significant things:

1. If the code has main() defined and runs with if __name__ = ‘__main__’: then this will not run with pweave, a simple main() outside of this block solves the problem although remember to comment one of these mains out before you run the untangled version else it will run twice 😉
2. pweave has several arguments and if you run python that also takes arguments then both arguments will be passed to both programs, inevitably causing argument errors.  For this example, I moved the MNist command line arguments to a config file.

Once I was happy that all four versions of the files ran without issue, I cleared the machine and did some timed runs.  As I wasn’t writing a paper and just needed indicative timing, I ran each version three times.  Nothing else was running on the machine ³, and the load was verified zero before each run.

To answer my own questions regarding performance:

1. RAM: Compared to the raw python, the pweaved code added 53Mb to virtual memory and 25Mb to physical memory used.  This appeared to be resilient to the complexity of the latex document on both GPU and CPU
2. CPU:  When the MNist example was running with the CPU option, there was no noticeable increase in usage, the system was maxed out for all examples.  When running as GPU, there was also no noticeable increase in CPU, even though the system had plenty of resources.
3. GPU:  Only relevant for the second set of runs on GPU, there was also no noticeable impact with running GPU intense code within pweave.
4. Time to train models:  with the three runs of each markup I did, the processing time varied by a few seconds within each type, although the average for each were all within 1 second of each other for a minimum run time of 5 minutes, showing nothing significant here.

So, while not as scientific as I’d like, this shows that pweave will hog a small amount of RAM, but otherwise shouldn’t impact the standard python programs.  This makes sense, as pweave generates the .tex file and processing that is a separate task.  I imagine that different set ups could give different results, particularly if there is much more complex interaction between the LaTeX and python than even my major mark-up document provided, although for my purposes it’s nice to know that there are no significant performance overheads to literate programming.  I’m looking forward to this being used more widely in the company.