On Sunday, 15 May 2022 19:22:43 CEST Carsten Agger wrote:

I wrote a rant against the idolization of Lean and Toyota among certain DevOps people. Originally it was the synopsis for a more thorough article which I don't have the time to write, so instead I present only the conclusions. Maybe of interest to people here involved in programming (or union work and working conditions, for that matter) since DevOps is everywhere these days and has its good sides too.

https://blogs.fsfe.org/agger/2022/05/12/devops-inspiration-from-toyota-produ...

I saw this and, being partial to a good rant myself, considered commenting on it. I think the adoption of Lean techniques and related methodologies and practices, already controversial in organisational settings where people might find their jobs incorporating elements of those techniques on the insistence of management, has been undertaken rather uncritically (or cynically) not just in DevOps but in software development and maintenance generally.

Naturally, there will always be good reasons for evaluating the way people work and deliver results in other industries, particularly when those industries appear to be better at managing projects and producing products within time and budget constraints. And when people introduce elements of practices from outside the software industry, skepticism is usually countered with an insistence that lessons need to be learned (which I generally agree with) combined with suggestions that skeptics don't want to change their ways (which I obviously do not agree with).

Often, people have to accept such changes as a complete package, having their reservations downplayed or mischaracterised, and this is how you can easily end up with situations where good and useful techniques become packaged up with less good and potentially harmful practices, perhaps even escorting some other kind of agenda through the door. If you find yourself identifying and resisting that broader agenda, it is entirely possible that you might risk being categorised as "unprofessional" precisely because of a deliberate conflation of practices, incentives and organisational politics.

It is interesting to consider how things have changed over the years, what improvements have taken place, and what kinds of problems or problematic trends have emerged. I think back to my consulting days a couple of decades ago and it is fair to say that some things have moved on. Some of those things are technology and tooling improvements, whereas others are social or behavioural.

To take an example of a clear improvement in a technological form, people were using tools like CVS to manage their source code back in my consulting era, but few people were using things like branches, partly because tools like CVS offered a rather incoherent experience with a lot of potential for mistakes and confusion. When distributed version control systems became more prominent, the industry gradually adopted them and started to leverage the benefits they brought with them, thus changing development habits, mostly for the better.

Meanwhile, to consider practices that were deficient back in that era, I remember the project I worked on having to deliver documentation on how the software deliverables were to be installed, with a Word document being painstakingly crafted, handed over to a bunch of other consultants (from the same company), who then copy-pasted or retyped commands from the document, handing it all back for revision if something didn't work. At that time, it was obvious (but not to the rather backward project management) that the proper way of delivering the system was through automation, with documentation supporting that automation.

One can argue that the whole consolidation of development and operations into DevOps builds on the realisation that developers cannot merely do their thing and then hand it over to other people to roll out. Developers should not be tempted into thinking that they operate in an ideal environment, although they are usually disabused of such thoughts at university. However, one can also argue that the two domains may require different skills and abilities and that combining them dilutes an individual's necessary focus on two potentially different activities, and it also excludes other necessary activities.

One prevalent trend that remains a source of conflict within the Free Software realm (and elsewhere) is that of emphasising container-based deployment (Docker, Flatpak, Snap, and so on) and seeking to bypass traditional software distributions. We have all surely heard it said that distributions provide packages that are "too old" whose maintainers "get in the way" and that other ways are needed of getting the absolutely latest stuff. This narrative plays into the whole DevOps culture: the developer has to get their stuff out there, has been given or has taken on this task, and nothing should get in the way.

The problem with this empowerment of the "ops-conscious" developer is that it eliminates any kind of role that seeks to preserve a coherent deployment environment: the job a distribution package maintainer might do where someone might consider whether the software is deployed correctly, has the appropriate level of privileges, interacts with other software harmoniously, and so on. Somehow, DevOps must absorb these concerns, but instead they are increasingly neglected or denigrated, as noted above.

Consequently, the industry has evolved in the absurd direction of giving every application its own container or virtual environment despite running in a multiuser system that already has plenty of facilities for isolating applications and restricting privileges (although I would accept that those facilities might need some adaptation). In effect, it almost seems that developers now expect to get their own virtual mainframe for their so very important applications.

Even though there are benefits in having the ability to "spin up" virtual machines on demand, and having been exposed to things like OpenStack I don't deny that it can be fairly convenient, tools need to be developed and provided to manage all these machines if everything gets its own "private mainframe". Coordination or orchestration of these new-style units of computing also becomes necessary. I remember tedious meetings with the customer in a consultancy era project about which network ports were required by our software, presumably to be scribbled into a Word document. There are certainly arguments to be made for capturing all of this and integrating it using technology, but I have reservations about the stack of complexity being built.

Perversely, as all this complexity spills out, presumably to be managed by big companies with big datacentres, the measures to manage complexity are arguably being neglected at the developer level. As developers clamour for the latest latest, and to be able to roll it out at any time, all those university-taught software engineering practices like compartmentalisation, defining stable interfaces, and so on, have been abandoned in favour of continual system-wide changes and pervasive refactoring, mitigated by the magic merge in people's version control systems.

And where traditional software distributions have been rejected, developers tend to seek similar levels of convenience elsewhere. It appears that this is sometimes just a matter of pulling container images from Docker Hub or some equivalent, or it is a matter of pulling down packages from the likes of CondaForge, or even pulling down images containing packages from these non- traditional distributions. Frequently, these alternative sources leverage traditional distributions, but don't expect anyone advocating these alternatives to acknowledge their hypocrisy. And as people rush to package for maximum instant convenience, don't expect distribution-level quality control, either.

People could potentially mitigate development chaos by laying down stable components in their systems that do not need instant access to the latest updates from upstream development repositories, and this would then lessen the need for highly customised software stacks and deployment environments. But this would need a mindset change, not only among developers but also in the industry at large. After all, developers are not acting as they do purely because they want to: it is because the nature of the work has changed, and it is because something has to give when they need to get the work done.

This is where I belatedly return to the social aspects of introducing potentially inappropriate methodologies to the software industry. We have all seen waves of "rationalisation" wash into the industry over the years. When developer salaries were seen as too generous, outsourcing and offshoring pushed many of the supposedly interchangeable developer jobs to low-cost venues with the idea that the grunt work could be done on the cheap, albeit managed by high-salary consultants. Such trends neglected well-established observations that good communication is essential in software projects, and so quality and productivity suffered as a result.

What we arguably see now is a form of redefinition of the roles within the industry, consolidating them into a kind of singular role where the occupant is required to be "responsive" to whatever the demands of the job are, where such demands are deliberately diverse so as to save the costs and the bother of having specialists, and also to permit those occupants to be interchangeable. Not that the motivation for this latter concern is necessarily benevolent, either: while it is good for knowledge to be shared in an organisation, the motivation for having redundancy in an organisation is often the potential for making people redundant entirely (that is, letting them go).

When I pursued a computer science education, I had hoped to find meaningful and rewarding work in the industry, but the kind of "responsive" roles we see now are effectively shallowing out the nature of the work. People are increasingly expected to deliver direct-to-user improvements on a continuous basis and to immediately respond to problems on an "everything has top priority" basis. Wiping out distinctions between "development" and "production" or "pre-release" and "release" might have ridden the industry of tortuous "beta" phases and software that has not been given enough user exposure, but it risks eliminating stability for both users and those who have written the software.

A significant risk for the industry is the effect this has on endeavours that require a sustained development focus without constantly being obliged to tweak something or other for the supposed benefit of users. Some will respond to such concerns by claiming that there are plenty of good-enough solutions out there already and that people should take the consumerist approach of picking and choosing, mixing and matching, and just give up on any significant new (or humanely pursued) development effort. All this does, however, is see people pile more and more stuff on top of the existing stuff - a heap of dubious complexity - to try and do what they need to get done.

Another risk for the industry is a kind of concentration of power and influence in organisations that are able to dedicate people to making solutions that the rest of the industry, in its constantly hassled state, end up having to adopt. Where does that leave individuals, developers or normal end-users, having to effectively "take it or leave it" with regard to the software they may obtain, whether it is Free Software or not?

I get the feeling that an upper echelon of the industry would quite like to see the rest of the industry as nothing more than virtual warehouse workers, scurrying around with their genius-designed products, bundling them up on a just-in-time basis and grateful for the privilege of doing so. This vision has not been delivered on previous occasions, but the one thing you can guarantee is that the last time somebody tried, it wasn't the final time.

Paul