When developing software it makes sense to be able to work on local files, while the source code should be served from a controlled environment (a container) to prevent pollution of the developer workstation.

In this article I will describe the evolution of a development workflow for deploying applications on OpenShift. The ultimate goal is to make it possible to maximize dev/prod parity, while minimizing the idle time in the change/test cycle.

OpenShift Source-To-Image makes it really easy to publish source code through container technology. Testing the resulting containers in OpenShift is easily done by using either oc cluster up or Minishift, but with the release of OpenShift 4 the first method becomes obsolete. This means that Minishift now is the go-to method for providing a local OpenShift environment for developers.

Note that I will use Python as the language of choice in this article, but a similar strategy can be followed when using other interpreted languages (Ruby, PHP, Perl). When using (byte) compiled languages (Java, GoLang), the build requirement for those makes using this strategy much harder to use.

Traditional local development

When using Python as the language, local development is typically done using virtualenv and the Python modules being used are then defined in requirements.txt. This happens to be the file used by the Python autodetection for OpenShift s2i, so the example repository makes it easy to activate a local environment with the correct Python requirements.

The following commands provide you with a local clone of the example code and a Python virtualenv using the default Python version from your workstation:

$ git clone https://github.com/pjoomen/hellopythonapp.git
Cloning into 'hellopythonapp'...
remote: Enumerating objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
Unpacking objects: 100% (32/32), done.
$ cd hellopythonapp
hellopythonapp$ virtualenv venv
New python executable in /Users/pjoomen/hellopythonapp/venv/bin/python2.7
Also creating executable in /Users/pjoomen/hellopythonapp/venv/bin/python
Installing setuptools, pip, wheel...done.
hellopythonapp$ source venv/bin/activate
(venv) hellopythonapp$ pip install -r requirements.txt
Collecting gunicorn (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Collecting Flask (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting Jinja2>=2.10 (from Flask->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from Flask->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Werkzeug>=0.14 (from Flask->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->Flask->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/cd/52/927263d9cf66a12e05c5caef43ee203bd92355e9a321552d2b8c4aee5f1e/MarkupSafe-1.1.0-cp27-cp27m-macosx_10_6_intel.whl
Installing collected packages: gunicorn, MarkupSafe, Jinja2, itsdangerous, Werkzeug, click, Flask
Successfully installed Flask-1.0.2 Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 click-7.0 gunicorn-19.9.0 itsdangerous-1.1.0

We can now start a local (debug) webserver:

(venv) hellopythonapp$ python wsgi.py
 * Serving Flask app "wsgi" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 318-601-219

Run the following in a separate terminal to check the code using the curl command (or use your preferred webbrowser):

$ curl
Hello Python World!

Using this setup, changes to the application (wsgi.py) are directly visible.

A new version of the application can be deployed to an OpenShift environment – without actually committing code – using:

hellopythonapp$ oc start-build hellopythonapp --from-dir=.

This method is easy to setup and maintain, but it has a few obvious disadvantages:

  1. The Python binary and the installed modules are not guaranteed to be identical as the ones being used in the containerized environments,
  2. The time-to-build for deployment to OpenShift is bound to become an annoying delay in the development cycle.

Adding Minishift into the mix

Minishift allows you to test your code easily against a locally running OpenShift, saving you for an (extra) external dependency. To install Minishift, follow the guidelines provided within the OKD documentation.

Now start your local Minishift installation:

$ minishift start
-- Starting profile 'minishift'
OpenShift server started.

The server is accessible via web console at:

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

Make sure you are logged in as developer and deploy and expose the sample application:

$ oc login -u developer
Logged into "" as "developer" using existing credentials.

You have one project on this server: "myproject"

Using project "myproject".
$ oc new-app https://github.com/pjoomen/hellopythonapp.git
--> Found image 7b379e8 (11 days old) in image stream "openshift/python" under tag "3.6" for "python"
--> Success
    Build scheduled, use 'oc logs -f bc/hellopythonapp' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/hellopythonapp'
    Run 'oc status' to view your app.
$ oc expose svc/hellopythonapp
route.route.openshift.io/hellopythonapp exposed
$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World!

Local changes can now be deployed by rebuilding the image from you working directory:

hellopythonapp$ oc start-build hellopythonapp -F --from-dir=.
Uploading directory "." as binary input for the build ...
Uploading finished
build.build.openshift.io/hellopythonapp-2 started
Receiving source from STDIN as archive ...
Push successful

This takes care of the aforementioned disadvantage of not having code parity, but the build-cycle is introducing a delay which introduces an idle-cycle on the part of the developer, and there is a limit on how often one can go for a cup of coffee.

Gunicorn and dynamic code changes

The following section is specific for Python code being served through gunicorn, but similar practices hold true for other languages (and their respective HTTP engines).

To enable gunicorn to reread changed code from the filesystem, we need to start gunicorn with a configuration file, containing the following:

debug = True
reload = True

Save this snippet in a file called config.py, put it into a ConfigMap and make it available to the DeploymentConfig for your application. Then enable gunicorn to use this configuration by adding an environment variable to the DeploymentConfig:

$ cat <<EOF > config.py
debug = True
reload = True
$ oc create configmap gunicorn --from-file=config.py
configmap/gunicorn created
$ oc set volumes dc/hellopythonapp --add --mount-path=/etc/gunicorn --type=configmap --configmap-name=gunicorn
info: Generated volume name: volume-tkqvv
deploymentconfig.apps.openshift.io/hellopythonapp volume updated
$ oc set env dc/hellopythonapp APP_CONFIG=/etc/gunicorn/config.py
deploymentconfig.apps.openshift.io/hellopythonapp updated

We now are running gunicorn with debugging and reloading enabled. Open up a terminal to be able to keep an eye on the logs:

$ oc logs dc/hellopythonapp -f
---> Serving application with gunicorn (wsgi) ...
[2018-11-09 15:04:31 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2018-11-09 15:04:31 +0000] [1] [INFO] Listening at: (1)
[2018-11-09 15:04:31 +0000] [1] [INFO] Using worker: sync
[2018-11-09 15:04:31 +0000] [31] [INFO] Booting worker with pid: 31
[2018-11-09 15:04:31 +0000] [33] [INFO] Booting worker with pid: 33
[2018-11-09 15:04:31 +0000] [35] [INFO] Booting worker with pid: 35
[2018-11-09 15:04:31 +0000] [38] [INFO] Booting worker with pid: 38

Copying code changes into a running container

The oc binary includes a rsync verb which can copy (changes in) files from your working directory to a running container. It can even watch for changes!

Note that there is no need to synchronize permissions between the source and the target. There is also no need to copy the .git and venv folders.

Open up terminal and start the rsync process:

$ cd hellopythonapp
hellopythonapp$ POD=$(oc get pods -l app=hellopythonapp -o custom-columns=NAME:.metadata.name --no-headers)
hellopythonapp$ oc rsync --watch --no-perms --exclude .git --exclude venv . ${POD}:/opt/app-root/src

In your original terminal execute the following:

hellopythonapp$ sed -i 's/Hello Python World/& v2/' wsgi.py
hellopythonapp$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World v2!

The first line makes a change to the source file using sed, which is then copied into the running container by the rsync process. The change will then become visible in the output of the curl command.

You can of course use your favorite editor and webbrowser to perform those tasks.

If you do this really quick, you might not see the change immediately as there is a delay in the change being picked up by both the rsync and the gunicorn processes.

Make sure you terminate the rsync process by pressing ^C in the terminal where it is running, before proceeding with the following section. You now can close this terminal.

Using a hostfolder for local source code

Although the rsync method described above is quite convenient, there is a way that is even more convenient, but a little bit more demanding to configure: minishift hostfolder.

When using this method you do not have to worry about having the rsync process running on your workstation. Once configured it is enough to have your Minishift VM up and running! This method is for all practical purposes identical to the way Vagrant makes local source code available using a shared mount.

To setup the hostfolder do the following from your terminal:

hellopythonapp$ minishift hostfolder add hellopythonapp --source $(pwd) --target /srv/hellopythonapp
hellopythonapp$ minishift hostfolder mount hellopythonapp
hellopythonapp$ minishift ssh -- ls /srv/hellopythonapp

For a container running under OpenShift to be able to read from this mountpoint we need to enable the virt_sandbox_use_fusefs SElinux boolean:

hellopythonapp$ minishift ssh -- sudo setsebool -P virt_sandbox_use_fusefs on
hellopythonapp$ minishift ssh -- getsebool virt_sandbox_use_fusefs
virt_sandbox_use_fusefs --> on

By default a container running in OpenShift is not allowed any access to the host it is running on. This can be fixed by changing the SecurityContextConstraints for the ServiceAccount running the container:

$ oc login -u system:admin
$ oc adm policy add-scc-to-user hostaccess -z default
$ oc login -u developer

Now the DeploymentConfig can be changed to use the mountpoint within the VM as the source for the code used by the application:

$ oc set volume dc/hellopythonapp --add --type hostPath --mount-path /opt/app-root/src --path /srv/hellopythonapp
info: Generated volume name: volume-kdcgs
deploymentconfig.apps.openshift.io/hellopythonapp volume updated

Now make another change and test to see if this change propogates:

$ sed -i 's/v2/v3/' wsgi.py
$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World v3!

Note that there still is a delay before the change is picked up by the gunicorn process.


The above gives three alternative ways for testing your local changes before committing and pushing them:

  1. Internal debug webserver for your environment,
  2. rsync verb from the oc binary,
  3. minishift hostfolder.

This allows for a more lenient code-build-test cycle and thereby preventing commit-log pollution, as well as having a more efficient workflow while working on new features, or fixing the incidental bug, for your application.

Pip Oomen

OpenShift Solution Manager at Redpill Linpro

Pip – aka. Pepijn for his Dutch countrymen – started in Redpill Linpro in 2012. He is a Red Hat Certified Instructor, Examiner and OpenShift Specialist and – as developer turned system-administrator – always looking for ways to 'Automate all the Things'."

Ansible/AWX network performance investigation

When we introduced the network configuration using Ansible and AWX at a customer, we gradually extended the configuration scope. Over time, more and more configuration got added into the configuration pool and this lead to longer and longer run-times for the playbooks.

While the job-execution got really simple by using AWX instead of the plain CLI method for Ansible, the time to finish drew heavily on that benefit.

A complete job-run over the network infrastructure took at least ... [continue reading]


Published on March 19, 2020

Multi-factor Authentication (MFA) for AWS CLI

Published on February 18, 2020