One of the first apps I set up on my Photon Docker host was Nextcloud. It’s an excellent solution for synchronizing and sharing files and comes as an official docker image. However, since I first set it up, I’ve been troubled that the clients start getting disconnected from the server after a short while (hours, a couple of days). The web UI works fine, but the clients on desktop and mobile show up grey with a 500 error.

Checking the logs on the container, I get something like this:

PHP Fatal error: Nesting level too deep - recursive dependency? in /var/www/html/lib/private/Log/ExceptionSerializer.php on line 215

Unfortunately, even after several new builds, this error seems to persist, so I guess I’ll record the workaround here, so I know how to reapply it if it’s overwritten in the future.

I’ll probably do this from within Portainer. If so, change the command to open the console to

/bin/bash -u 33

Then run this as a workaround in Circles:

./occ config:app:set circles route_to_circle --value ''

Then if needed, enable Circles again

./occ app:enable circles

This seems to have fixed the problem for me. At least it’s been working so far.

This is a follow-up to my post on setting up Photon OS on vSphere as a lightweight VM to run containers on. By the end of that post, I have the OS up and running with Docker Engine in place and enabled. This post will add simple, graphical container management to the host through Portainer.

You might ask, “Isn’t the point of containerized workloads, infrastructure as code, etc., that you can use the command line to manage your workloads?”. And you might be right. But right now, I’m not building dev environments; I’m bringing up relatively static components, prepackaged containers, that will deliver some sort of service in my network. And I won’t be deploying, destroying and redeploying these daily. And since my memory has always been terrible, and I have more than enough I need to remember from my actual work, I’d have to be looking up these things all the time. So sometimes, a friendly, intuitive GUI is just a better option. The nice thing here, of course, is that I can still do both, choosing whatever works best for me at any given time.

Portainer comes in two different flavours, the free Community Edition, and the commercial Business Edition. The Business Edition is available for free for fewer than 5 nodes; for now, I’m installing Community Edition, but I might look at the BE too eventually.

Because I want the host VM to be “disposable” so I can rebuild it and the contents for whatever reason, I needed somewhere to put the persistent data of the containers, including the Portainer management container. So I made an NFS share on my NAS called conda because it’s for container data, but I’m too lazy to type it out.

Since the minimal install of Photon OS that you get when deploying the OVA doesn’t include the nfs-utils package that I need to mount the share, the first step is to install that using tdnf:

tdnf install nfs-utils

With that done, I can make somewhere to mount the NFS share; mkdir/var/conda ought to do the trick. If my NAS has IP 198.51.0.10, this will accomplish what I want:

mount -t nfs 198.51.0.10:/conda /var/conda

Now we have somewhere persistent to store the configuration or user data of the containers, it’s time to deploy Portainer.

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /var/conda/portainer_data:/data portainer/portainer-ce:latest

This will pull the latest community edition of Portainer, and store the config data in /portainer_data on my NAS share, mounted in /var/conda on the VM. Once done, it should be accessible on port 9443 on the IP I assigned to the Photon OS instance. Set a password, and you’re up and running:

From here, you can deploy and manage both single containers, and you can do Docker Compose installs using what Portainer calls stacks. I’m not going to do an in-depth discussion on Portainer features, mainly since I just started using it, but their Youtube channel is an excellent source of info. Once I have stuff to discuss, I might do another post.

I’d like to highlight the Stacks feature, which is basically a UI for Docker Compose, where you can type your YAML right in their web editor, upload it as a file, or connect to a repository.

No one will believe me, but I was pretty surprised to find that VMware’s Photon OS seemed to be the perfect fit. Of course, we have it as the base of many VMware appliances, and it’s also commonly used as the OS inside the containers deployed in Tanzu Kubernetes Grid, such as the supervisor nodes. That it also comes as a standalone OS with the Docker Engine in place and ready to go hadn’t registered with me before. And, of course, it’s already optimized to run on a hypervisor with as small a footprint as possible.

This is primarily for my own reference to know what I did when I need to redo this; here’s the process.

First, go to the download page and grab the latest version. Since I’m deploying this on ESXi, I’m getting the OVA with virtual HW version 13, which VMware describes as a “Pre-installed minimal environment, customized for VMware hypervisor environments. These customizations include a highly sanitized and optimized kernel to give improved boot and runtime performance for containers and Linux applications.”. There’s also an ISO installer and images for other hypervisors and clouds. Some of the stuff here is covered in the installation docs, but some of it is not. Surprisingly.

Deploy the OVA as you would any other OVF/OVA. There’s not much to decide on during the deployment, but I make sure to set the disk to thin-provisioned as I’m starting to run out of space on my SSD datastore.

Once deployed, it’s set up with 1 vCPU and 2GB RAM. The disk footprint with thin provisioning is an impressive 670MB. I increase this to 2 CPUs and 4GB RAM and fire it up. As instructed, you need to change the root password from “changeme” on the first login.

Welcome to Photon 4.0 (x86_64) - Kernel 5.10.83-6.ph4-esx (ttyl)
photon-machine login: root
Password :
You are required to change your password immediately (administrator enforced). 

The next step is to set a static IP address. Find the name of your Ethernet interface:

root@photon-machine [~]# networkctl
IDX LINK TYPE     OPERATIONAL SETUP
1   lo   loopback carrier     unmanaged
2   ethO ether    routable    configured
2 links 1isted.

Then create a network configuration file and fill out the info for the name of the Ethernet interface:

root@photon-machine [~]# cat > /etc/systemd/network/10-static-en.network << "EOF"

>[Match]
>Name=eth0

>
>[Network]
>Address=198.51.0.2/24
>Gateway=198.51.0.1
>EOF

Change the permissions of the file

chmod 644 10-static-en.network

And apply the new network config

systemctl restart systemd-networkd

While we’re at it, we might as well set the hostname to keep things nice and tidy and help identify resources on the network:

hostnamectl set-hostname MyComputerName

Check that your VM has the static IP. Now it’s time to get it up to date and secure. First, see which packages are available:

tdnf check-update

Or, if you’re curious about the relevant security advisories:

tdnf updateinfo info

Then run the upgrade command to apply the patches

tdnf upgrade

OK, so the OS is running, connected to the internet and up to date. Let’s get Docker up and running. I initialize the docker engine:

 systemctl start docker

And then make sure it’ll run on boot:

 systemctl enable docker

Let’s check out what the status is

docker version

Looks great! I could start to deploy containers right now. But let’s get some management functionality up and running to make things easier. I think that’s suitable for a separate post; read on here.

At work, I’ve been involved in projects over the last few years to deliver a private cloud experience to our users. One of the cornerstones in the latest endeavour has been to provide a solid container platform based on Kubernetes. This is quite a challenging adjustment for someone who has done primarily IT operations around traditional workloads, first physical servers and later virtual machines but still running conventional operating systems like Linux and Windows.

Kubernetes and software-defined infrastructure require a different approach to IT operations and blur lines between traditional roles. It requires developers to take greater responsibility for security and infrastructure elements and network, storage and server admins to better understand how code is built, delivered, and updated. While I’ve written some basic web applications through the years, I certainly don’t consider myself a developer, and I’m struggling to fully grasp some of the concepts and ideas.

I’m a practical guy; I need to try stuff, really use it, to fully understand something genuinely new and different. So, therefore, I decided to bring containers into my home lab and bring up a Kubernetes environment that I can test stuff on.

Running a bunch of QNAP NASes at home, the easiest and quickest way to start tinkering with containers was to deploy them using the included Container Station, which offers the Docker engine underneath. The first application I brought up here was a Crashplan cloud backup application. The person who had been building QNAP native packages of Crashplan was retiring the project, and to back up the NAS content directly, the docker image was the best-supported option. This has been running for at least a year and works very well.

Since then, I’ve added two more docker containers to Containers Station on my primary NAS. But running all sorts of containers is not what I want to use the NAS for, so I’m looking for a more suitable platform even though it’s a reasonably well-specced unit with 4 cores and 32GB RAM.

So instead of the next natural step, which would be to get something to run Docker on, I took a leap and started looking for a way to run Kubernetes. Since we’re using Tanzu Kubernetes Grid on top of a vSphere platform at work, I tried to find something similar, but without all the requirements of TKGs, which would massively complicate my little lab. After a lot of tinkering, I managed to bring up a management cluster using Tanzu Community Edition. Then I realized I really needed some infrastructure components up before proceeding with a workload cluster, so I had to return to planning for a Docker environment after all.

Which probably qualifies for its own post…