DevOps inspiration from Toyota Production System and Lean considered harmful

[Paul Boddie]

On Sunday, 15 May 2022 19:22:43 CEST Carsten Agger wrote:
> I wrote a rant against the idolization of Lean and Toyota among certain
> DevOps people.
> Originally it was the synopsis for a more thorough article which I don't
> have the time to write, so instead I present only the conclusions. Maybe
> of interest to people here involved in programming (or union work and
> working conditions, for that matter) since DevOps is everywhere these
> days and has its good sides too.
> 
> https://blogs.fsfe.org/agger/2022/05/12/devops-inspiration-from-toyota-production-system-and-lean-considered-harmful/

I saw this and, being partial to a good rant myself, considered commenting on 
it. I think the adoption of Lean techniques and related methodologies and 
practices, already controversial in organisational settings where people might 
find their jobs incorporating elements of those techniques on the insistence 
of management, has been undertaken rather uncritically (or cynically) not just 
in DevOps but in software development and maintenance generally.

Naturally, there will always be good reasons for evaluating the way people 
work and deliver results in other industries, particularly when those 
industries appear to be better at managing projects and producing products 
within time and budget constraints. And when people introduce elements of 
practices from outside the software industry, skepticism is usually countered 
with an insistence that lessons need to be learned (which I generally agree 
with) combined with suggestions that skeptics don't want to change their ways 
(which I obviously do not agree with).

Often, people have to accept such changes as a complete package, having their 
reservations downplayed or mischaracterised, and this is how you can easily 
end up with situations where good and useful techniques become packaged up 
with less good and potentially harmful practices, perhaps even escorting some 
other kind of agenda through the door. If you find yourself identifying and 
resisting that broader agenda, it is entirely possible that you might risk 
being categorised as "unprofessional" precisely because of a deliberate 
conflation of practices, incentives and organisational politics.

It is interesting to consider how things have changed over the years, what 
improvements have taken place, and what kinds of problems or problematic 
trends have emerged. I think back to my consulting days a couple of decades 
ago and it is fair to say that some things have moved on. Some of those things 
are technology and tooling improvements, whereas others are social or 
behavioural.

To take an example of a clear improvement in a technological form, people were 
using tools like CVS to manage their source code back in my consulting era, 
but few people were using things like branches, partly because tools like CVS 
offered a rather incoherent experience with a lot of potential for mistakes 
and confusion. When distributed version control systems became more prominent, 
the industry gradually adopted them and started to leverage the benefits they 
brought with them, thus changing development habits, mostly for the better.

Meanwhile, to consider practices that were deficient back in that era, I 
remember the project I worked on having to deliver documentation on how the 
software deliverables were to be installed, with a Word document being 
painstakingly crafted, handed over to a bunch of other consultants (from the 
same company), who then copy-pasted or retyped commands from the document, 
handing it all back for revision if something didn't work. At that time, it 
was obvious (but not to the rather backward project management) that the 
proper way of delivering the system was through automation, with documentation 
supporting that automation.

One can argue that the whole consolidation of development and operations into 
DevOps builds on the realisation that developers cannot merely do their thing 
and then hand it over to other people to roll out. Developers should not be 
tempted into thinking that they operate in an ideal environment, although they 
are usually disabused of such thoughts at university. However, one can also 
argue that the two domains may require different skills and abilities and that 
combining them dilutes an individual's necessary focus on two potentially 
different activities, and it also excludes other necessary activities.

One prevalent trend that remains a source of conflict within the Free Software 
realm (and elsewhere) is that of emphasising container-based deployment 
(Docker, Flatpak, Snap, and so on) and seeking to bypass traditional software 
distributions. We have all surely heard it said that distributions provide 
packages that are "too old" whose maintainers "get in the way" and that other 
ways are needed of getting the absolutely latest stuff. This narrative plays 
into the whole DevOps culture: the developer has to get their stuff out there, 
has been given or has taken on this task, and nothing should get in the way.

The problem with this empowerment of the "ops-conscious" developer is that it 
eliminates any kind of role that seeks to preserve a coherent deployment 
environment: the job a distribution package maintainer might do where someone 
might consider whether the software is deployed correctly, has the appropriate 
level of privileges, interacts with other software harmoniously, and so on. 
Somehow, DevOps must absorb these concerns, but instead they are increasingly 
neglected or denigrated, as noted above.

Consequently, the industry has evolved in the absurd direction of giving every 
application its own container or virtual environment despite running in a 
multiuser system that already has plenty of facilities for isolating 
applications and restricting privileges (although I would accept that those 
facilities might need some adaptation). In effect, it almost seems that 
developers now expect to get their own virtual mainframe for their so very 
important applications.

Even though there are benefits in having the ability to "spin up" virtual 
machines on demand, and having been exposed to things like OpenStack I don't 
deny that it can be fairly convenient, tools need to be developed and provided 
to manage all these machines if everything gets its own "private mainframe". 
Coordination or orchestration of these new-style units of computing also 
becomes necessary. I remember tedious meetings with the customer in a 
consultancy era project about which network ports were required by our 
software, presumably to be scribbled into a Word document. There are certainly 
arguments to be made for capturing all of this and integrating it using 
technology, but I have reservations about the stack of complexity being built.

Perversely, as all this complexity spills out, presumably to be managed by big 
companies with big datacentres, the measures to manage complexity are arguably 
being neglected at the developer level. As developers clamour for the latest 
latest, and to be able to roll it out at any time, all those university-taught 
software engineering practices like compartmentalisation, defining stable 
interfaces, and so on, have been abandoned in favour of continual system-wide 
changes and pervasive refactoring, mitigated by the magic merge in people's 
version control systems.

And where traditional software distributions have been rejected, developers 
tend to seek similar levels of convenience elsewhere. It appears that this is 
sometimes just a matter of pulling container images from Docker Hub or some 
equivalent, or it is a matter of pulling down packages from the likes of 
CondaForge, or even pulling down images containing packages from these non-
traditional distributions. Frequently, these alternative sources leverage 
traditional distributions, but don't expect anyone advocating these 
alternatives to acknowledge their hypocrisy. And as people rush to package for 
maximum instant convenience, don't expect distribution-level quality control, 
either.

People could potentially mitigate development chaos by laying down stable 
components in their systems that do not need instant access to the latest 
updates from upstream development repositories, and this would then lessen the 
need for highly customised software stacks and deployment environments. But 
this would need a mindset change, not only among developers but also in the 
industry at large. After all, developers are not acting as they do purely 
because they want to: it is because the nature of the work has changed, and it 
is because something has to give when they need to get the work done.

This is where I belatedly return to the social aspects of introducing 
potentially inappropriate methodologies to the software industry. We have all 
seen waves of "rationalisation" wash into the industry over the years. When 
developer salaries were seen as too generous, outsourcing and offshoring 
pushed many of the supposedly interchangeable developer jobs to low-cost 
venues with the idea that the grunt work could be done on the cheap, albeit 
managed by high-salary consultants. Such trends neglected well-established 
observations that good communication is essential in software projects, and so 
quality and productivity suffered as a result.

What we arguably see now is a form of redefinition of the roles within the 
industry, consolidating them into a kind of singular role where the occupant 
is required to be "responsive" to whatever the demands of the job are, where 
such demands are deliberately diverse so as to save the costs and the bother 
of having specialists, and also to permit those occupants to be 
interchangeable. Not that the motivation for this latter concern is 
necessarily benevolent, either: while it is good for knowledge to be shared in 
an organisation, the motivation for having redundancy in an organisation is 
often the potential for making people redundant entirely (that is, letting 
them go).

When I pursued a computer science education, I had hoped to find meaningful 
and rewarding work in the industry, but the kind of "responsive" roles we see 
now are effectively shallowing out the nature of the work. People are 
increasingly expected to deliver direct-to-user improvements on a continuous 
basis and to immediately respond to problems on an "everything has top 
priority" basis. Wiping out distinctions between "development" and 
"production" or "pre-release" and "release" might have ridden the industry of 
tortuous "beta" phases and software that has not been given enough user 
exposure, but it risks eliminating stability for both users and those who have 
written the software.

A significant risk for the industry is the effect this has on endeavours that 
require a sustained development focus without constantly being obliged to 
tweak something or other for the supposed benefit of users. Some will respond 
to such concerns by claiming that there are plenty of good-enough solutions 
out there already and that people should take the consumerist approach of 
picking and choosing, mixing and matching, and just give up on any significant 
new (or humanely pursued) development effort. All this does, however, is see 
people pile more and more stuff on top of the existing stuff - a heap of 
dubious complexity - to try and do what they need to get done.

Another risk for the industry is a kind of concentration of power and 
influence in organisations that are able to dedicate people to making 
solutions that the rest of the industry, in its constantly hassled state, end 
up having to adopt. Where does that leave individuals, developers or normal 
end-users, having to effectively "take it or leave it" with regard to the 
software they may obtain, whether it is Free Software or not?

I get the feeling that an upper echelon of the industry would quite like to 
see the rest of the industry as nothing more than virtual warehouse workers, 
scurrying around with their genius-designed products, bundling them up on a 
just-in-time basis and grateful for the privilege of doing so. This vision has 
not been delivered on previous occasions, but the one thing you can guarantee 
is that the last time somebody tried, it wasn't the final time.

Paul