For the past few months, the growth on EngineeredTruth has been the same. I've been…
…why state programming sucks.
This is a story about events 20 years ago that have changed my opinion about the PLC automation field that I was entering, whereafter I decided to enter the (then rapidly developing) IT networking field.
I have an electrical engineering degree and I'm a network administrator for more than a decade. I originally intended to enter the automation industry, but events in the late 1990's changed my mind about the profession. The information provided here is to account the events that happened, and might benefit you if you consider entering the PLC programming field. Take in consideration that economics change and things change during the years and the issues I've experienced may not apply to you in this field.
I started off in the software programming industry first as a C + SCADA programmer for about 1.5 years. When I finished my studies the recession of the early 1990's were finally starting to wear off and most of my former fellow students have already found a job, so did I. The conditions provided by my first employer were not the best, but the experience gained was considered very good for future employment, so most left for better grounds within a year. I thought I would be better off in the PLC automation industry, because PLC programming was what I had done in my thesis and I thought this would be easier. It proved to be a mistake.
I found a new job as a PLC programmer. Two smaller projects were first presented to me, first additions to an newly implemented existing Siemens S7 installation, and then a fully new Siemens S5 installation. After a month I was joined with a senior automation engineer and had to implement an S5 program in a big project. The specifications of the PLC were provided by another project engineering company who was implementing the physical infrastructure (piping, fitting, vessels) via subcontracting. I was not told the specifics of the contract of this project of any other project with this external engineering firm. They've send us the specifications of the PLC program every 3, 4 days in big envelops with bundles of papers filled with lines of conditions and actuators that had to be steered, all written down in listings of finite-state-machine states.
The way the PLC program was structured was very simple: the PLC program contains a dozen sequential programs that has to be executed in step wise fashion. Only one of these sequential programs was allowed to be executed at once, and each step in these sequential programs had its own step number. There are programming blocks in Siemens S5, called SB's (*Step Block, or originally called Schritt Baustein), but this way of programming was not used. Instead a custom programming scheme where a whole FB (*Function Block) designated for every step was used. Every step or FB, contained 3 networks: one network for the next step conditional to jump the sequential program to the next step, one network where a Set-bit is responsible for setting actuators high and one network where a Reset-bit is responsible for resetting actuators. A small central step-control FB is used to jump to the right step, and basically all what it does is jump with an indirect “B MW 200, SPA FB 0” instruction using a step number loaded in ACCU1 to the correct step FB. When a jump to the following step was requested via the next step conditional, the actuators in the current step FB were reset using the Reset bit, the step counter is increased, then in the same PLC scan cycle, the next step FB was called with the Set-bit so that unchanged values in outputs don't toggle their actuators. A sequential program could exist out of between 10 steps = 10 FB's to 20 steps = 20 FB's, there were about 15 sequential programs and in each step / FB there were around 70 actuators to be set or reset. A quick calculation of these shows there were 15x15x70 = around 16000 actuators to be programmed in around 220 FB's. The full listing of all the conditionals for step changing and the conditionals for setting and/or resetting of the actuators were provided in bundles of listings provided on sheets of papers, knitted together with a stapler, and I had to copy all the listings of these actuators into FB's.
After 2 months programming we discovered lots of redundancies were present in the specifications as provided, redundancies we couldn't have foreseen because only a single sequential program was delivered every 3 to 4 days, and it took about 1.5 months before we could extrapolate that these redundancies were there all over the whole PLC program, so rewriting would render all previous work void. We had all those actuators that weren't updated that were replicated over all the steps. Above that we actually found that individual steps on one sequential program closely mimics other steps in another sequential program. The programs were just variations of each other. All these steps were virtual identical, and the whole PLC program could be implemented a lot smaller if we had known this in advance. Some shortcuts were implemented, but more as an afterthought.
I had to enter all the programming in actual S5 AWL code. All decisions of laying out the steps and allocating inputs, outputs and conditions was already done by the external project engineering firm. I had no input: just tedious copying listings of countless conditions and actuators provided on +1000 pages of text. This extremely boring job took between 3.5 to 4 months to finish, including SCADA screen development.
Now, as some of you might think, “Who works in this way? Why doesn't employ the external project engineering form employ their own PLC engineer?” This question occurred me as well: the only rational I can give is that the external project engineering company has no resources to provide full 24×7 support needed by companies, and is outsourcing support to us, and we have to implement these programs in a way we could troubleshoot in a very instrumental way without consideration of the specifics of the actual program. This was not as what is told to me, it was told to me the reason they work like this is when a support call for an support engineer is issued by a company, the support engineer doesn't have to spend time understanding the PLC program. All this way of working was not told to me in advance when I applied for the job.
When entering of all this programming code was finished, the department head and the senior PLC engineer were to implement the PLC program on-site at the factory. I was designated to implement a similar program for a smaller similar installation, not only in entering a similar but shorter finite-state-machine states Iisting, this time only around 70 steps and a smaller amount of actuators where involved. I was asked to implement it in a similar way, with the same FB's as in the project that was done in the previous job. Because of the smaller scale of the program I could enter the provided sheets in a few weeks and start implementing it on-site with an (female) engineer of the external project engineering company.
On-site, first a small issue emerged with overlapping MB with a communication FB used for a control panel and the MB used inside the central step-control FB: the steps didn't worked as intended when the control panel was enabled. This was simple to solve, just move the MB of the step-control FB to another region. From the MB 250-253 to the MB 240-243 region or similar, still functional identical, but I had to update the corresponding conditional jump in each step FB, took some time.
Then a more serious problem emerged. The central step control FB showed a problem: when a new sequential program was started after the previous program was finished, the PLC jumped to the last step of the previous program instead of the first step of the newly started program. Was this standard central step control FB not supposed to be bug-free? This was puzzling. Generated a cross-reference of the whole PLC program, I started to verifying the individual merkers of the central step control FB to verify there was no conflict. Debugged the code, the step control FB AWL code showed to be non-intuitive as it combined both word operations, word shift operation, a bolean word using a mask operation, and individual bit operations, to make it very compact. As an ultimate test I disabled all the rest of the program except this FB, and it proved that the problem was due to the step control FB itself and not caused by some of my code. Then I made a call explaining the issue to the senior engineer on-site at Leuven. He was very surprised hearing of the issue, and replied this central control was used everywhere and never had any issues with it..
During debugging the block I discovered the merkers of the central step FW were not reset after ending the last step of a program, causing next program to start in the last step. Why? I compared my implementation with the previous project that I used as reference, and no mistakes were made, everything was identical. To continuing the testing the implementation I have to resort to a conditional reset after the central step control FB of each sequential program. This would allow testing, and in the empty periods where the project manager of the external project company was doing their own business allowing me to troubleshoot the code of the central step control FB.
A week before the deadline when most troubleshooting was done the project manager of the external engineering company made the request to implement a manual mode into the PLC program: it must be possible to enter a code on the display, select motors, pumps, valves on the display, set these in manual mode and toggle the output on those actuators. This was a big change, to implement this I had to move all the output actuators outside the individual steps and combine it with a conditional automatic/manual mode bit before output of the bit to the actual actuator. Changing this would be extremely time consuming. Of course I was objecting but the project manager kept insisting this manual mode had to be entered. I was thinking to make another call to the senior engineer.
Why I didn't? I was thinking… this whole programming method didn't made any sense. The central step control FB was super compact written, and while you might use extreme smart coding in PLC to save space and allow smaller models of PLC's to be used, it didn't made send to save space having unreadable piece of code and then waste a lot more space to redundant code. I had to enter all these lists of conditionals and actuators for months which was frustrating, time consuming and after all proved to be inflexible for changes, right in the first project I had to do. The central step control FB didn't work as intended which was very odd because if it was used everywhere it should have worked. The request for the manual mode just a week before deadline was totally off the hook. How didn't the project manager could haven't known this request would imply changing the programming if this way of programming was used everywhere? Today I know I have to ask to write these requested changes down in paper, so there can be no dispute at all, but remember I was fresh from college, I had no clue how to handle this. But back to the question: why didn't I made the call to the senior project manager? Think again… all this 5 months of entering listings of boring conditions and actuators… do I want to do this do this for a career?
I remember the PLC courses I took during my college years. We had all these PLC programming exercises. Because I already did some PLC programming in my high school I was aware there is a easy way to avoid difficult ladder logic programming: just define states using merkers, set the next merker when the condition for the following step was met, and reset the previous step merker using the inverted merker, easy done. Our group was the fastest to implement all these exercices. When the teacher discovered I was using this method she wrote special assignment for me I couldn't use this state programming method, to get me out of this habit of programming! Now, 2 years later, I was here on-site with this enormous volume of copied stack of code where there is no room far change, because if I have to change something in one step FB, I have to change it in all step FB's to maintain consistency… ironic isn't it? This shows how enormous inefficient and inflexible state based programming really is. I had to make a decision.
I evaluated the possibilities to change it to allow integrating a manual mode without rewriting everything. There was a solution: you can rewrite an output in the current scan cycle again, because the buffered outputs are only after the end of the scan cycle written to the actual outputs. It's bad design, because now you have to keep track of which output in the PLC program is actual written. This could prevent rewriting all the step FB's, but there was a catch: I had to avoid the program to jump to the next step while running in manual mode. Not running the central step control FB was no option because there were outputs dependent on inputs inside the step FB's: this was a Mealy type state machine, not a Moore. Two other options remained: adding the manual condition on the next step in every step FB, but to do this I had to change every step FB. Thus remained rewriting the code of the central step control FB just to avoid stepping during in manual mode was the last option that was left. This took a day, and after implementing I hadn't any problem with it.
Then came the testing the latest code, because I had to add this manual mode code, and we had only a week left, we had to sit through testing and troubleshooting sessions until 3 a 4 o' clock each night. It involved code writing sessions at night in probably the coldest hotel room in the Ardennes near Orval in Belgium in the winter. Note that if the manual mode was not requested by project manager of the external engineering company, the project would have been finished last week. Sunday afternoon, the day before the deadline, it was ready, it was fully programmed, implemented and tested in little over 5 weeks.
Monday morning the department head of the company I was working for called me, to ask me about the state of the project. I answered that we had a lot of issues, but the project was fully implemented and working. He requested me to be on site tomorrow. The next day he wasn't on site, which wondered me. During that day I had another call with the project manager of the external firm, nothing alarming was said during that call.
That evening, I was requested at the HR department. The HR has said they have heard bad things about me, and said that I was very opinionated. Afterwards I was asking myself these questions: Didn't I complained to the project manager of the external company the requested manual mode was not requested in advance and because of this I had to rewrite the step control FB? Didn't I explained to her I had to change the step control FB for allowing the manual mode to function? Why didn't the department head asked my take on what has happened? Why was the deadline placed just a day before the end of my 6 month trail period? Why didn't the department head assigned a project manager on this project in the first place?
Some observations: I'm pretty sure the department head didn't read my code because: I've made my decision to rewrote this step control FB under high time pressure and the changes could be reversed in a heartbeat. I'm literally talking about 1 page of AWL code! I didn't inform him and all changes were on my laptop only until the last day he wasn't in the office. In addition of that I remember an instance the department head reviewed my code very shallowly where he spend max 10 seconds on and another instance he handled this 5 week job, he told he could do this in a weekend. This was definite the worst employer I have ever worked in my career. He provided with no mentor ship and just used me to enter thousands of lines of boring code.
This events made me to reconsider my career. Most likely I was only hired just to do the boring entry stuff to offload work. Clearly the real intention of this way of programming was to made it very easy to replace someone on a dime and this was very easy in PLC programming. The automation field is a field that is very conservative: that main concern companies have with PLC technology is their reliability and availability. Companies will rather stick to an expensive PLC technology (hint: Siemens) that has shown to be reliable, and to be available not only in replacement parts, but also I number of programmers. A lot of PLC programmers are just technicians, and are just needed for maintenance of existing infrastructure. This would imply that I have to compete with a lot of people willing to work at a lower wage. This was obvious not the best environment to work in, so I decided to change to another field.
I decided to change to change to the networking field. The Internet was just starting to take off and become popular, and this was an exciting new field with new technologies which were becoming dominant: TCP/IP, Cisco, Web browsers, firewalls, Linux, Windows NT, proxy servers, Exchange, email.
I applied for a starter job at a translation company, where I have managed it's network and servers top-to-bottom. Within 2 weeks I had a new job and with better pay. The day I solicited for the job, I still remember the moment I entered the entrance hall of the building: I saw the building was shared by the company with another company. At the left was the logo of the translation companies name on it, but to the right…
I kid you not… (think about it, there are more than 100000 companies in our country!)
…to the right was the logo of the external engineering company I was working for last 5 months!
So imagine one year later, I met the project manager I was working with last year. I asked her what happened with the project, was all my code scrapped? She said: “no no, there were really great ideas in the code!”
Now years later when I reexamine the automation industry, I see there are these industrial PC's where a real-time kernel is running on a Windows machine, much faster than the PLC's. But when I examine the programming environment the same programming technology still applies. Some improvements were implemented (like IEC 61131-3, object oriented programming, etc…) But when I tried the screen programming I remembered the same dread when I was working on SCADA: how inflexible it was to handle the GUI elements.
Today, when I examine the professional IT world, there are all these technologies out there changing the field in a very structured way: massive leaf-spine data centers, virtualization, convergence, deduplication, docker, software defined networking, micro-segmentation, software appliances, Puppet, cloud networking and the list goes on and on. In programming Python is moving forward and is pushing programming concepts as functional programming like lambda, list comprehensions, generators, decorators, etc. New database concepts like NoSQL have emerged. Very fast development and delivery processes are used. Next generation networking technologies IoT will change the landscape even more: now you will get all these appliances that will have wireless networking build-in. And then there is there also data science and machine learning…
Compared with the IT industry, the PLC industry is lacking new programming concepts, which could change the industry tremendously. The reason is obvious, the need to maintain and service the existing installations has a higher priority than trying some new (flangled) technology and risking having no support when this technology fails. It is still a very vertical industry. So no, I am not planning to go back to PLC programming again, but for those who are willing to enter this industry, think again: “software is eating the world”. How long this continues, I don't know.
- SB = Step Block / Baustein, is the finite state programming blocks of the Siemens PLC environment
- Actuator = An output driving a lamp, coil, motor, contactor
- AWL = Anweisungsliste, the assembler language of Siemens PLC
- FB = Function Block / Baustein, is a numbered function, provides AWL
- Network = inside a PLC program a network is a organizational division, you may compare it with a page or paragraph in text.
- MB = Merker Byte, is a 8-bit memory unit in Siemens PLC
- OB1 = Organization Block / Baustein, is the main program loop in the Siemens PLC, called 100's times per second
- SPA = Jump to label/Block instruction
- ACCU1/2 = two main registers used in Siemens PLC, used for (double)word operations
- Indirect instruction = a value is preloaded into ACCU1 and replaces the 0 value of the next instruction, eg B MW 200, SPA FB 0, means jump to FB block number where the value is located in MW 200
Since I wrote this text I have asked myself at what point I could have figured out the micromanagement was all about control and not about support management. The first odd thing is the project engineer reverse engineering the state diagrams because he could have asked the external engineering firm these diagrams, but this only indicates to a migration. The first actual hint I received was just before I departed to the facilities was in a last call with the manager. After explaining the things I had to do, he suddenly jumped over to another topic. After this project he would continuing working with me migrating his state based S5 programming scheme to the new S7 programming language. While he was explaining this I asked myself “why are you talking about this now?”. The S7 language is mostly backward compatible, so he could migrate his step control block of only 20 lines within 10 minutes. Now unfortunately I forgot this brief conversation while testing. In hindsight it was pretty clear now how insignificant this migration effort would have been, and a direct indication that he just wanted me to believe me he wanted to continue employing me after the project. It happened very early in my career, and at that time I assumed events presented to me were normal. Now, with more years on my resume I can say my first intuitions are mostly correct, and my self-doubt was often what did betrayed me, otherwise stated: when things do not make sense, there is a reason for it.
Now the events did learned me a life lesson. Look at the events: did the manager really benefitted from this tactic? actually no: he not only fired the engineer working on his project, the most likely outcome of this was the project manager taking up over the role and having to “correct” everything. I know as a fact the outcome of this, he decided to leave the company the very next month after I was fired. So the project manager decided to leave just looking at my work. Coincidence? I think not. For me, it started a new path in my career, where I never assumed thing anymore as they are presented to me.