IPython Dev Meeting - Winter 2014

  |   Source

Some recap from my perspective of IPython dev meeting in Berkeley. Less code than last time, more tired. More complex problem to solve.

Bi-Annual Meeting

As you might know (or not), with the Sloan Grant came some funding to allow a reunion of the all IPython core-dev and active participants twice a year; last July we had our first 5 days meeting in Berkley, so it was time to meet again and discuss the future of IPython.

In this post I'll try to sum up what append during the week to bring you up to speed as, unlike everyweek, we did not do our discussion in public on google hangout. I hope this might explain our low presence on the net for the last five days.

For the technical detail, I was the only one that was flewn from Paris on last saturday (11th), I was welcommed by Thomas Kluyver (that I finally met after 2 years), Jess Hamrick and Aaron Culich who helpped me to stay awake on the first day in front of Beers and burgers; I fought jetlag in San-Francisco on Sunday. Then the hard part started, you put between 8 and 10 people in a close room with coffe from 9:30am to 6pm for 5 days and mix.

Usually we did code only from 8am to 9am while having breakfast, hence the low apparent productivity (except from Jon and Min that might probably have their clone at home).

We had extra people from last time, in the name of Kyle Kelley from Rackspace, who is responsible for nbconvert deploy, and Thomas Kluyver who finished his PhD last summer and joined the IPython team full time.

The two exception were wednesday where the room was open from 4pm to 6pm for anybody to comme and say hello, as well as thurstday evening, were we had dinner with other people related to IPython around berkeley.

Human After All

Of course not every minute was productive, in particular Brian had the inside of his screen cleanned and I'll see if I can get permission from Thomas to post a video of him learning to drive (Musée des Méchaniques, near Fisherman Wharf in SF). We also learned that Brides could thow cats.

We spoke of Markov Chain which lead us to examine the example of training on the King James Bible and Structure and Interpretation of Computer Programs aka King James Programming :

I am the LORD when I have preached to others, I myself should be a function f between a and x.

Memorable Quote were Said:

As well as

Just add a var that = this; (On Python code)

IPython 2.0

We are slightly late on the scedule, but some things had to be discussed. It was a good thing to be all in one room. Expect a release of 2.0 beta in 4 to 6 week at best.

[Modal UI] was merge 2 weeks ago, and will be refined, we know some people are not happy with it, and we are aware it is not perfect. We were starting to hit the limit of what we could do with browser and had to go modal to develop the notebook. We will still be limited by the browser capability and fight with the dom focusing of elements; so some compromise had and will have to be made. We thought a lot of what we wanted, we are happy with it and need 4 to 6 six weeks of the modal UI in the wild to get user feedback and polish rough corners. So if you have comment, complaint we would be happy to hear from you.

We also a huge discussion of Interactive Widget, from a tecnical point of view, User eXperience, and code review. We are pushing some functionality from 2.0 to later, it will not prevent you from dooing anything but might need little more code in the kernel. Expect Jon Pull request to be merged in a week or so, then 4 more weeks to have testing from users. If you are adventurous, grab Jon PR today and test it! Jason Grout made a lot of good feedback, advice and technical review.

Unexpected Changes

We had an unexpected discussion about library like mpld3 and static widget (@Jakevdp we love your work and congrats BTW) which lead to a small digression onf 1 day and a half about dynamic content (on nbviewer | live notebook) and consequence on security as well as good practice. I'll try to quickly sum-up the problem.

Notebook allow arbitrary content execution, but you don't want arbitrary code to be executed. So we need to walk the fine line between what is code that user want to run, from code that user would prefer not to run.

As of IPython 1.0, viewing a notebook is equivalent of running code. With the possibility we keep adding to IPython, at some point notebook could wreak havock on your machines, and we want to avoid making things like document macro that could be malicious.

(If I get some time later I would be happy to make a talk that show some of the things that would be doable)

In the meantime we like to avoid aving a user experience like UAC where in the end you just click Allow without pondering the consequences. Keep in mind that you have to view notebook as program, that run as yourself and that we cannot guaranty full security when you open notebook on your local machine, but we will do our best.

That beeing said, as some content like HTML/Javascript/svg might present security they might not be show in some circonstance when you open a notebook, so from 2.0 (probably) and oward we will (try to) :

  • Prevent rendering markdown if it contains any Javascript
  • HTML/JS/svg in output
    • will not be show on load
    • unless you were the one to have fully executed the notebook for the last time.

This will not apply (for the time beeing) on nbviewer, and we have no reason to disable it until we add authentication to nbviewer.

To do so, starting with 2.0 notebooks files will be signed (as in cryptographycally signed) on save with IPython web frontend (Aka webnotebook). This is of course not mandatory for a notebook to be runned or loaded. It does not change the notebook format, and do not affect existing kernels. Though if you have a custom frontend (or program that generate notebook) I would advise for you to follow the same practice.

From a user perspective, this means that if I download the example notebook mpld3 on my machine, it will not show as on nbviewer up until you have run all cells at least once in once session. Once done it will not affect you.

If you work a lot across multiple machine and like your notebook to be recognized everywhere, we will provide more technical details later on where the key(s) are stored by default.

Public/Private Keys

I am personnaly really happy about the signature mechanisme we are putting in place. It might add noise to versionnig system (but you can always have a hook that clean it). I believe it could be tightly coupled to authorship of total or part of a notebook.

We did quite some discussion on the current mechanisme, almost come to blows, and add several iterations before comming to that. We suppose security is not perfect and we love people to review that. Personally I will try to hook that into Public/Private key pairs to be able to have a ring of trust of notebook.

I hope also that at some point it might help in versionnig to know when a notebook has merges (once the merge tool is written) and might not reflect what a full re-execusion would do.

This does not prevent you from considering that every notebook you load could be dangerous, and probably do more damages than you think.

IPython 1.2

We will have a 1.2 release.

IPython 2.x

We will (probably) try to release 2.x release regularly (every month). Bug and doc fixes only.

IPython 3.0

Not everything planned for 2.0 was done, some many things have been just pushed to next release, we new about it but it is always hard to see the number of things that have to be done.

As for myself, I wasn't assigned to any issues, indead I'll have to start writing my dissertation but I'm thrilled to see what is comming. We have not made a detailed plan yet but here is what we are thinking about:

  • Static Widgets

    • runs through all combinations of widgets and sends results to the notebook.
    • Where do we store all te data without making 2Gb notebooks ?
    • Make it compatible with nbviewer ?

  • Full, integrated support for non-Python kernels

    • Launch IPython Once, and ability to select "Open this notebook with (IJulia|IRuby|IHaskell|...) etc.
    • Simpler way for kernels to provide "Configurability" for IPython.

  • Diff/merge of notebooks

    • Greg Wilson have student workign on that
    • We will try also

  • UI to browse filesytem/edit non-ipynb text-files

  • Push nbviewer forward
  • Starting Multiuser-support
    • learn node.js

Rackspace

Nbviewer have moved to rackspace a few weeks ago, and we continuing to move things over there. Thanks to Kyle Kelley and Jessy Noller, we know have ability to deploy nbviewer on Rackspace servers usig salt-stack (with a small discount for open source project). Movign DNS controle over to Rackspace will allow us to semi-automatically deploy PRs to test and have clean load balancing when doing a deploy not for the service to drop user when a new version go into production.

We (and this mean Kyle) are working hard to document this completly from A to Z and make as much as possible public so that you could replicate it locally. Right now we still have some things private because of Auth keys.

After That...

4.0... And we do not know, Sloan Grant funding end on 31 Dec 2014. So if you or your entreprise have used of IPython consider helping and feel free to write to us.

Bye from thousand of feet above Atlanic Ocean, it's 5:30 am were I'm heading to, and 8:30pm where I come from. Next IPython Meeting will be in roughly 6 month, mark the date and come say hello. Otherwise search us at PyCon, SciPy, EuroSciPy (..etc) we have Stickers and Flyers!

As usual my english is not perfect, so PR are welcommed, More details topic later.