needs more opera: a Placissimo update

I recently posted [http://blog.humaneguitarist.org/2018/09/03/easy-command-line-and-restful-interfaces-to-python-functions-with-a-plac-intensifier/] about creating a simple way for Python developers to use their command-line Python applications over HTTP. In this post, I'm going to flesh that out a little more. Background At work, we ended up with a web application for something, but it seemed to me our glue code between our Python scripts and our HTML was exceedingly unnecessary. In other words, it was re-creating the argument parsing logic that our scripts already did and the code around logging and sending those logs to websockets is simply too much to maintain. It's also just a distraction from our core requirements. So it seemed to me that what is needed, agnostic of my work stuff by the way, was a simple way to bridge Python command line scripts with an HTTP interface. That way, one should only have to work on their UI and not one writing a one-off backend to support talking to an underlying script. Given the nature of my work, I can only work 11 months in a row, so I've been off all October. So I've been working on that idea. I'm calling it Placissimo given that I'm saying it's an 'intensifier' for Plac [https://micheles.github.io/plac/]. Instead of working on the code this morning as I sit in the wonderful Brew Coffee Bar [https://www.facebook.com/BrewDTR/] in Raleigh, I'll write about how it works. I've got a cold or some such nonsense and I'm not the mood to do development. Besides, this post will help me get started on documentation. How It Works Let's say I have a simple program, my_scripy.py , that uses Plac to create a command line interface. I'll run it from the command line and use the -h help flag. > py -3 .\my_script.py -h usage: my_script.py [-h] [-iso] path Prints/Returns a tuple: the path contents and the current time. positional arguments: path path to a folder optional arguments: -h, --help show this help message and exit -iso, --iso use ISO format Now, I'll pass it the current folder. > py -3 .\my_script.py . INFO:__main__:Hi from: .\my_script.py INFO:root:child here (['my_child.py', 'my_script.py', '__pycache__'], 'Mon Oct 29 11:54:58 2018') You can see it printed the tuple at the end, as well as a few logging statements, one from itself and one from an imported script. What you can't see if that it took 3 seconds to complete ... because I added a 3 second delay because I need that to test something else that's not important to get into for now. Let's run the script again, this time with the -iso flag so that the time format is altered. > py -3 .\my_script.py . -iso INFO:__main__:Hi from: .\my_script.py INFO:root:child here (['my_child.py', 'my_script.py', '__pycache__'], '2018-10-29T11:57:36.131385') Now, let's look at the annotated part of the main function in my_script.py as well as how the call to Plac would look like if I was using it directly. def main(path: ("path to a folder"), iso: ("use ISO format", "flag", "iso")=False,): """Prints/Returns a tuple: the path contents and the current time.""" # ... some code here ... if __name__ == "__main__": import plac plac.call(main) Now, let's see what happens when I use placissimo instead a la: if __name__ == "__main__": import placissimo placissimo.call(main) The command line examples above work exactly the same, so I won't repeat them. What's different? Well, now I can use the --servissimo command (I've been watching a lot of opera lately) and something different happens. I'll actually add the -h help flag first ... > py -3 .\my_script.py --servissimo -h usage: my_script.py --servissimo [-h] [-allow-websocket] [-allow-get] [-port 8080] [-index-path None] [-filesystem-path None] Server options. optional arguments: -h, --help show this help message and exit -allow-websocket enable "/websocket" endpoint -allow-get enable GET access -port 8080 port number to use -index-path None path to HTML file for "/" endpoint -filesystem-path None path to parent directory for "/filesystem" endpoint Now, in addition to a regular command line interface, there's also an HTTP interface option and all I had to change was two lines - and one was an import statement. Of course, I could have just made one change a la: import placissimo as plac Server Details The server backend uses Tornado [https://www.tornadoweb.org/en/stable/index.html]. Tornado does everything I need and more. It also has a really well thought out API which is more than I can say for the other packages I looked into. Starting the Server I'll launch the server using the defaults a la: py -3 .\my_script.py --servissimo Using POST I can access the script via POST and the /api endpoint like so: curl --data "path=." http://localhost:8080/api I get this response. { "servissimo_0": { "caller": "my_script.py.main", "start_time": "2018-10-29T12:11:38.323433" } } This tells me that servissimo_0 is the name for the task I've started as a thread. I can check its status like so: curl --data "name=servissimo_0" http://localhost:8080/tasks Here's the metadata about that task: { "servissimo_0": { "caller": "my_script.py.main", "start_time": "2018-10-29T12:11:38.323433", "end_time": "2018-10-29T12:11:41.330387", "running": false, "done": true, "result": [["my_child.py", "my_script.py", "__pycache__"], "Mon Oct 29 12:11:41 2018"], "exception": null } } It finished without errors about 3 seconds after it started. That makes sense, given the hard-coded 3 second delay I mentioned earlier. Using GET Say I want to make GET requests a la: curl "http://localhost:8080/api?.&iso=True" All I would have to do is use the -allow-get option when launching the server. Rendering HTML If I had an HTML file I wanted to render, I could pass that in while launching the server with the following addition: -index-path="..\placissimo\index.html" . There will be some default objects that one will be able to use while rendering the HTML file using Tornado's template syntax [https://www.tornadoweb.org/en/stable/template.html]. For example, this: <dd>Starts a task by calling <code>{{ calling_module }}.{{ funk.__name__ }}()</code> will get rendered as: <dd>Starts a task by calling <code>my_script.main()</code> Moreover, if I want to pass in an object to render I can do something like this: placissimo.call(main, render_object=[1,2]) I can reference that in HTML like this: <ul> {% for item in render_object %} <li>{{ item }}</li> {% end %} </ul> Of course, that renders as expected: <ul> <li>1</li> <li>2</li> </ul> I'm probably also going to add an option that will convert the object to JSON ahead of time if the user specifies - and if it's a JSON compatible object. That would allow one to render their object as a JavaScript variable from the start a la: var placissimo__render_object = {{ render_object }}; Python Logging Statements To access logging statements, I can pass in the -allow-websockets option when calling the server. Clients connect via the /websocket endpoint. All logging gets sent to client websocket connections as JSON. Furthermore, the logs include the thread name. So, using the example above, our task name and the thread name for that task is servissimo_0 . This would allow one's JavaScript code to filter in only logs emanating from that task. Browsing Files and Folders The command line allows one to traverse the filesystem and get some metadata about file and folders. So, I thought placissimo needs to as well. For security reasons one has to pass in the starting point of a given file system a la: -filesystem-path="." . The server isn't supposed to display the contents of any folder that is an ancestor or a sibling of the path specified. If you try to, the server returns an HTTP 403 Forbidden Error . If a non-folder is passed to the optional path parameter, an HTTP 422 Unprocessable Entity Error is returned. Here are some examples of file and folder browsing: //# curl "http://localhost:8080/filesystem" { "my_child.py": { "container": ".", "is_folder": false, "path": "my_child.py", "size_in_bytes": 87, "creation_date": "2018-10-02T12:28:41.992999" }, "my_script.py": { "container": ".", "is_folder": false, "path": "my_script.py", "size_in_bytes": 1143, "creation_date": "2018-10-15T14:46:24.095074" }, "__pycache__": { "container": ".", "is_folder": true, "path": "__pycache__", "size_in_bytes": null, "creation_date": "2018-10-15T16:03:20.328304" } } // # curl "http://localhost:8080/filesystem?path=__pycache__" { "foo": { "container": "__pycache__", "is_folder": true, "path": "__pycache__/foo", "size_in_bytes": null, "creation_date": "2018-10-18T15:00:40.306949" }, "my_child.cpython-36.pyc": { "container": "__pycache__", "is_folder": false, "path": "__pycache__/my_child.cpython-36.pyc", "size_in_bytes": 338, "creation_date": "2018-10-29T11:43:43.687382" } } // # curl "http://localhost:8080/filesystem?path=__pycache__/foo" { "1.txt": { "container": "__pycache__\\foo", "is_folder": false, "path": "__pycache__/foo/1.txt", "size_in_bytes": 0, "creation_date": "2018-10-18T15:00:47.353890" } } What's Left to Do? Besides addressing some known bugs, proofreading code, writing documentation, and writing some simple test scripts and examples, I'm also going to add an option for secure cookie access, but in the end placissimo is not going to be an option for anything dealing with sensitive data. When I've got something cleaned up and complete, I'll throw it on GitHub. Conclusion I do think there's something to creating simple web applications on top of command line Python scripts - between the basic access to the script via HTTP, thread handling, optional logging access via websockets, and optional filesystem information. It's just the way we did it at work doesn't scale and it came with increased complexity and cost. Which probably means much of that code will get thrown away at some point. By the way ... What if your Python command line script uses something other than Plac? I'll guess it's not going to be hard to change to Plac or to write a Plac-based wrapper script around the existing script. I do recommend looking into Plac. It really removes having to waste time writing lots of code to provide a command line interface and it enforces a measure of good UX - for example, it doesn't allow required optional parameters. And, yes dammit, UX is important for command line programs, too. "Servissimo" ... really? Finally, in case anyone (that means nobody) actually reads this, I'll answer the burning question ... Is there a way to call the server if you don't speak Italian? Sì ... placissimo.call(main, server_name="launchServerPlease") Then you can just do: > py -3 .\my_script.py --launchServerPlease -h usage: my_script.py --launchServerPlease [-h] [-allow-websocket] [-allow-get] [-port 8080] [-index-path None] [-filesystem-path None] ... All your task and thread names will then start with the completely boring string prefix "launchServerPlease". That's all for now. Ciao. PS: Just hope I don't change placissimo.call() to placissimo.callissimo() .