Dirk Meister

Tip of the Week articles at FlashArray

11 Oct 2020

This week marks the half-year anniversary of FlashArray Engineering Tip of the Weeks.

These tips are short mails sharing knowledge about the programming languages we use and our software infrastructure. The idea is blatantly copied from Abseil’s Tip of the Weeks. As they say, imitation is the highest form of flattery.

Pure Storage’s FlashArray Engineering team has grown a lot in recent years. We rely a lot of code reviews for knowledge transfer, but relying on code reviews for knowledge transfer doesn’t scale well after a certain point. So, a year ago we started thinking about how to revamp knowledge transfer practices so that the software engineering organization can strive in its second decade.

Knowledge transfer isn’t getting easier by our style of C++, which differs in style and substance from general practice. From our own concurrency system including different locking primitives over our own smart pointers to our own allocators, guidelines you find about C++ might not always apply at FlashArray.

At the same time, it is getting more important in a time of Work-From-Home because traditional means of knowledge transfer are strained by the situation.

So we started to collect items that tech leads explain again and again and publish them in form of these tips. The tips follow a specific form, which I think are essential for the idea to be successful and not to become just another unread mail in the inbox:

What are the rules?

  • About coding. We would veto everything management-related or even tooling related. To stay relevant and trusted by engineers, we have to stay focused. On the other hand, all coding related tips are on the table. We have tips about C++, Java, and Python. If we have tips are pure C++ and tips covering our own infrastructure or how we apply C++ at Pure.
  • 200 - 500 words. Engineers should be able to read this in a few minutes.
  • Reflect what the consensus is. A TotW is not a place to publish new ideas. It should always reflect what the consensus about what should be done is. Each article is reviewed by other enginers and tech leads. They are good to keep you honest. I once wrote an article about “East Const” and it was vetoed because while I assumed “Prefer East Const” reflects the consensus at Pure, it actually didn’t.
  • Weekly: If you only sent a tip monthly, you cannot cover sufficient ground. If you sent out daily, people would stop reading. It would also become very stressful to produce that amount of articles. Weekly proved to be a great compromise.
  • Relevancy: Each tip needs to teach at least 30% of the audience something new and relevant. A tip shouldn’t only cover what is taught in our new hire training because then it would be boring for most readers. On the other hand, explaining an infrastructure that most engineers will never use in their careers at Pure is also not a great topic. We need to be relevant for the engineers if TotW wants to stay relevant. We cannot force anybody to read them or to apply them in practice.
  • Not a replacement for other forms of knowledge transfer: TotW is not a replacement for our new hire training and certainly not a replacement for documentation.
  • Tone: The tone is important. While our style is pretty down to earth, I am not good at injecting jokes into my writing, we always try to not preach from the pulpit. It is always important to treat engineers with respect. Nearly all guidelines have exceptions. We hardly use the word “Never” and each time we did there was a long discussion before we went with it.

In addition, each TotW gets a number and a permanent link. The hope is that engineers not only read them and apply them but actively quote them in code reviews. For that, we write the tips in simple Markdown and render the text into HTML to email them and into Confluence Wiki Markup language for the permanent link. The numbering scheme (also taken from Abseil) also makes referring to tips in reviews and discussion easier.

Topics

What topics did we cover in the last fix months? Most of them do not make sense outside of Pure, so I replaced the exact titles with a short summary.

  • TotW/0: Introduction into FA Tip of the Week
  • TotW/1: When noexcept matters? As we do not use exceptions, in general, we usually do not use noexcept. However, sometimes noexcept matters for performance, e.g. in move constructors.
  • TotW/2-3: Introduction into our own specialized string types. Due to some arcane technical detail of how our database is implemented, we have our own string types and they have a few pitfalls.
  • TotW/4: Assertions in unit tests.
  • TotW/5-6: Introduction into our error injection system in unit tests.
  • TotW/7: C++ and the auto keyword
  • TotW/8: Mark single-parameter constructors explicit by default
  • TotW/9: A tip on how to write good error messages
  • TotW/10: Introduction into our specialized vector-like data structure
  • TotW/11: A tip concerning w.r.t. to writing medium and large scale tests in Python
  • TotW/12: Parameterized unit tests
  • TotW/13: Tips about a strange detail about our concurrency system. In hindsight, this was the least relevant tip we published.
  • TotW/14: C++17’s return value optimization
  • TotW/15: An article about the principles of physical design and why we use struct more often then class. This is one of the questions that we get a lot from new hires because it looks very odd, but there are actually good reasons behind our approach to physical design.
  • TotW/16: Tips about our approach to configure the system.
  • TotW/17: Tips about primitive and boxed types in Java
  • TotW/18: Introduction into the Abseil’s btree_map. We do not use (yet?) Abseil in general. However, we made the btree_map available and are working on making the Swiss table available, too.
  • TotW/19: Why you never should use using namespace in header files.
  • TotW/20: Tips when and when to not use the Curiously Recurring Template Pattern (CRTP)
  • TotW/21: Further tips about configuration the system focussed on unit tests
  • TotW/22: Tips about high-level principles of highly concurrent systems. This is the first article in a bi-weekly series about concurrency.
  • TotW/23: Tips about when and how to comment code
  • TotW/24: Tips about own mutex implementation
  • TotW/25: Tips about our background job system
  • TotW/26: Tips about our concurrent hash table

We purposefully switch topics and even the difficultly so that even when one article wasn’t interesting to an engineer, there is a good chance that they next one might.

Conclusion

The feedback has been very positive. The tips are actively cited in code reviewers, which was the main goal. The tips have become an integrated piece into our training and knowledge transfer. One thing I learned how difficult it is to acquire more authors. Many senior engineers are active with reviewing articles (often multiple times) and ideas. However, getting people to write articles is hard. Of 26 articles only four have been written by other authors. My goal for the next 26 articles is that I write less than half of them by doubling down on outreach and outright bartering.

I actually enjoy writing from the tips to documentation, but (as you have figured out by now) I am not very good at it. While I live in the US, English is my second language. I wish we could have a technical writer, e.g. our user documentation team, supporting the project.

Summary

It is an approach to knowledge transfer with medium effort, focussed on the engineer and provides results. It doesn’t require tooling changes or workflow changes. It doesn’t require any special infrastructure. A mailing list and a Wiki page would do it.

Related posts