While performing vROps Tenant App 2.6.2 upgrade in my lab I have encounter the following error:
Failed to install updates(Error while running installation tests).
Quick check of the /opt/vmware/var/log/vami/updatecli.log shows that the appliance is running out of free space on the root / partition.
24/02/2022 15:01:34 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/32/test_command
Verifying packages…
Preparing packages…
installing package tenant-app-8.6.0-18724818.noarch needs 1231MB on the / filesystem
24/02/2022 15:01:41 [ERROR] Failed with exit code 56576
The reason why this is happening is that the tenant app runs as a docker container and the older versions have not been purged. I can see in my particular case I have above 7 GB of docker images on the filesystem:
root@tenantapp [ /var/lib/docker/overlay2 ]# du -h -d 0 7.5G . /var/lib/docker/overlay2 ]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE vmware/vrops-vcd-tenant-app-db-cassandra 2.6.2-19235005 057345d369fd 5 weeks ago 634MB vmware/vrops-vcd-tenant-app-db-cassandra latest 057345d369fd 5 weeks ago 634MB vmware/vrops-vcd-tenant-app-ui 2.6.2-19235005 4e90d15d3116 5 weeks ago 396MB vmware/vrops-vcd-tenant-app-ui latest 4e90d15d3116 5 weeks ago 396MB vmware/vrops-vcd-tenant-app-plugin 2.6.2-19235004 de4cb469fb65 5 weeks ago 309MB vmware/vrops-vcd-tenant-app-plugin latest de4cb469fb65 5 weeks ago 309MB vmware/vrops-vcd-tenant-app-db-cassandra 2.6.1-18326916 3b7ef9b0c10c 7 months ago 597MB vmware/vrops-vcd-tenant-app-ui 2.6.1-18326916 b66e34b5d59b 7 months ago 368MB vmware/vrops-vcd-tenant-app-plugin 2.6.1-18326915 f97bc56c3d61 7 months ago 286MB vmware/vrops-vcd-tenant-app-db-cassandra 2.6.0-17922920 0d5eb9de1cb7 10 months ago 581MB vmware/vrops-vcd-tenant-app-ui 2.6.0-17922920 3ffdeee597ca 10 months ago 354MB vmware/vrops-vcd-tenant-app-plugin 2.6.0-17922919 b23bd4eb6a2d 10 months ago 268MB vmware/vrops-vcd-tenant-app-db-cassandra 2.5.0-16990343 af72dbf16623 16 months ago 536MB vmware/vrops-vcd-tenant-app-ui 2.5.0-16990343 62b09bd2a0a2 16 months ago 252MB vmware/vrops-vcd-tenant-app-plugin 2.5.0-16941875 1217f67efd9d 17 months ago 190MB vmware/vrops-vcd-tenant-app-db-cassandra 2.4.0-15996298 a0d906a5cc5a 22 months ago 494MB vmware/vrops-vcd-tenant-app-ui 2.4.0-15996298 777fe7bc0c1f 22 months ago 240MB vmware/vrops-vcd-tenant-app-plugin 2.4.0-15996297 b85369dbf061 22 months ago 180MB vmware/vrops-vcd-tenant-app-db-cassandra 2.3.0-14826918 556121e468da 2 years ago 466MB vmware/vrops-vcd-tenant-app-ui 2.3.0-14826918 eb77c613e9ad 2 years ago 224MB vmware/vrops-vcd-tenant-app-plugin 2.3.0-14826917 e598e66d4818 2 years ago 158MB
After checking with Tenant App engineering, the problem has been fixed in the newest (8.6.1) version that does purge the old images upon successful upgrade. But if you hit the issue you will need to cleanup the old images with the follwing command:
docker image rm -f <image ID>
BTW if you delete wrong images you can always recreate them with the following commands:
docker load -i /opt/vmware/app/vrops-vcd-tenant-app-ui.tar.gz
docker load -i /opt/vmware/db/vrops-vcd-tenant-app-db-cassandra.tar.gz
docker load -i /opt/vmware/plugin/vrops-vcd-tenant-app-plugin.tar.gz
Update 3/3/2022
I have noticed the Self Health page on Tenant App admin UI in the Support section did not display any running services even though they (the docker containers) were running properly. After checking with engineering this can be fixed by modifying permissions of docker.sock file with:
chmod 666 /run/docker.sock
