Cleaning source code with sed

Recently a ticket came across my desk to remove references to Google Plus from a set of email templates our team maintains. With dozens of email templates, I wanted to stretch my command line muscles and see if I could do it all at once.

macOS includes sed, the stream editor, and for this project I’m going to use a very simple set of commands.

Delete lines that match

d is for delete. To use this command, you have a regex, followed by the letter d. It will delete any line that matches.

For example:

sed -i.bak '/googlePlusIcon/d' someFile.txt

Substitute lines

s is for substitute. Think of it like “find and replace” except we are going to find something everywhere. There are 2 parts, a regex to match, and a regex

sed -i.bak 's/googlePlusIcon/twitterIcon/g' someFile.txt

Doing multiple files at once

You can do multiple files at once by combining find with sed. Find syntax is as follows:

find path expression

path is the scope where find will… find things. expression is a set of options to control what we match. For example, -name is to match on the filename.

Another use of expressions is to do some operation. -exec is for executing another command. In this case we want to execute sed. So we type out the sed command from above, except with a few changes. There are curly braces {} which tell find to put the filename of the match in there. There is also the \; which tells find where the -exec expression is ending.

find . -name '*.txt' -exec sed -i.bak 's/googlePlusIcon/twitterIcon/g' {} \;

One quirk about find + sed is that it likes to add newlines to the end of your files. If you are using source control like git, this may cause the diff to be slightly larger than you expected.

Java on Mac Cheat sheet

Install AdoptOpenJDK following instructions for macOS. The instructions don’t say where to install it, I suggest /Library/Java/JavaVirtualMachines. This will get you the built-in mac behavior.

Built-in mac behavior

Older versions of Java created a Java button in System Preferences. It will not show Java 11, but it will show older versions. In my case, I had some old projects using Java 8.

The other built-in is the java_home utility.

List available versions

alias javas_avail='/usr/libexec/java_home -V'

Example output:

Matching Java Virtual Machines (2):
11.0.1, x86_64: "OpenJDK 11.0.1" /Library/Java/JavaVirtualMachines/jdk-11.0.1.jdk/Contents/Home
1.8.0_161, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home

Switch to a specific version

java_home will also allow you to switch to a given version. For example

$ export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
$ java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

If you find yourself switching versions a lot, create an alias for it.

Docker cheat sheet

docker run = new container (specify image name)

docker exec = run a command in an existing container

How to turn a docker run command into an interactive shell:

-it --entrypoint=/bin/sh

Note, you have to add that before the name of the image.

Cleanup images

To filter down the images and only clean those matching some-text, use:

docker images | grep some-text | awk '{ print $1 ":" $2}' | xargs docker rmi

Cleanup orphaned images

Sometimes images are orphaned, for example, if you tagged an image as latest, and then built a new image with that same tag. They will show up as <none> in docker images. They can be cleaned up using:

docker rmi $(docker images -f "dangling=true" -q)

List images by size

docker images --format '{{.Size}}\t{{.Repository}}' | sort -hr

Two types of chronic anxiety from “A Failure of Nerve”

I just finished reading A Failure of Nerve: Leadership in the Age of the Quick Fix, by Edwin Friedman. It’s a book about leadership in the #2018TechReadingChallenge.

Friedman’s key concept in the book is chronic anxiety, which is the foremost challenge to leaders. Leaders have “a failure of nerve” when the fail to rise above the anxiety that surrounds them.

Since we all are surrounded by anxiety, we all have the opportunity to become leaders. One of Friedman’s catch phrases is “from parents to presidents,” reflecting that anyone can be a leader, whether you are a leader of a family, or of an entire country. In my day job, I’m a leader by being a senior software engineer, since I am responsible for delivering value, minimizing risk, as well as mentoring the junior developers on the team.

The desire for a quick fix is the first example of anxiety. It occurs when we reactively look to solve a symptom, rather than being thoughtful and addressing the underlying problem. We look for techniques and methods, without realizing that the last 6 methods we tried did not solve the problem. Maybe the quick fix is converting the entire organization to scrum, without realizing that the underlying relationships of your organization are toxic.

A leader is a person who realizes that methods and techniques won’t solve problems. Instead, it’s more about what you do after you make the decision.

Reactivity is the second example of anxiety. Reactivity in leadership is the person who blames others when there is a problem, or who has the same knee-jerk reaction every time. It’s the person who blows up when something unexpected happens. It’s me, when I take it out on a coworker whose work didn’t meet my expectations.

The well-differentiated leader is less reactive. “Those who are less reactive are more self-contained, less blaming, more imaginative, less anxious, and more responsible. When they do seek help, they generally can hear suggestions well, offer less resistance to suggestions for change, and treat their consultant as a coach rather than a savior. Such an approach emphasizes strength rather than weakness, accountability rather than blame, taking responsibility for self rather than feeling for others.” (p. 164)

In a software development context, the well-differentiated leader is the person who is able to maintain their professionalism in the mist of an emergency bug fix. Or the person who is able to clarify expectations and help the team deliver, even when the priorities of the organization seem to be constantly changing. Or a person who handles interruptions to their work with grace, even when it is the 3rd interruption that day.

Anxiety is not just an individual challenge, it is a challenge for organizations and societies at large. Anxiety is a driving force in the emotional processes that permeate our interactions one another. Unchecked, these emotional processes will stifle our imagination. Regardless of the emotions themselves, these processes have a way of being passed down from generation to generation, and so the greatest challenge of a leader is to see and overcome the processes surrounding him or her.

Instead of our imagination being stifled, we must take responsibility for our own being and destiny, which Friedman calls being a well-differentiated leader. This is not a matter of finding the right techniques or having a self-improvement checklist, and so in this sense, the book is not a typical self-help book. Rather, it aligns more with leadership books such as Ordering Your Private World, and Leading with a Limp. It also reminded me of The 7 Habits of Highly Effective People, which I did a review of a few years ago (part 1) (part 2).

What is software architecture?

In my post about Event sourcing and CQRS, I touched on software architecture, but assumed it was already defined. I’d like to dive into that a bit more based on some recent reading.

My definition of software architecture is providing structure that allows productive, effective software development to occur.

Provides structure means having the appropriate technology, practices, culture, and organization in place to allow flourishing. As a software developer I tend to think of the technology (the code) first, but it is certainly not the only thing needed to have effective software architecture. More on that later.

Productive, effective software development means producing software that meets the needs of the business. There are 2 ways that software meets the needs of the business:

  • What the software does
  • How it does it

Or to use more formal terminology:

  • Functional requirements
  • Non-functional requirements

The functional requirements are the business behaviors that provide value to the customer. So for example, if we are creating software for a library, we need the ability to check out books, check in books, and keep a library catalog. Each of these has several behaviors behind it, and there are probably other behaviors I am missing.

The non-functional requirements are also known as the “-ilities” that describe things about the software besides its function. For example, testability is how easy the software is to test. Maintainability is how easy the software is to maintain. There are many others:

  • Evolvability
  • Scalability
  • Security
  • Extensibility
  • Feasibility
  • Reliability
  • Deployability

Besides the above, non-functional requirements might relate to the skill level of developers required to maintain the system, or the structure of the development teams that maintain it. (e.g. the presence of an off-shore development team)

The central claim of software architecture is that you can implement functional requirements in any architecture, but different architectural styles have different non-functional characteristics. What do I mean by that? They provide software structures that emphasize some characteristics at the expense of others. This is a classic engineering trade-off. The job of the software architect is to select an appropriate software structure that matches the non-functional requirements, which should also match the business expectations.

My term “software structure” would overlap with the term architectural pattern, which I first heard about from Mark Richards. Some examples of architectural patterns include:

  • Layered architecture
  • Microkernel architecture
  • Pipes and filters architecture
  • Service oriented architecture
  • Event driven architecture
  • Space-based architecture
  • Microservices

Ah yes, microservices, the buzzword of the last few years. I don’t have time to go into microservices in this blog post, but check out Jimmy Bogard’s Microservices FAQ for more info.

I mentioned earlier that there are different aspects of providing structure. In addition to architectural patterns, which affect the code (technology), there are:

  • organizational patterns (e.g. are teams organized by business capability or by similar job roles? What is the mapping between products to teams?)
  • practices (such as TDD or Extreme programming)
  • culture (probably another blog post altogether)

Ideally all four aspects of structure work together to accomplish the desired outcome. (I’m not sure of the origin of this list of 4 items came from, if anybody knows, leave a note in the comments.)

Code generation

Code generation is a common tool in software. Usually it sits behind the scenes, just a layer of automation translating from one programming language to another. But sometimes it becomes useful to build your own code generator. In a project at work, we’ve implemented a code generator based off a DSL (domain specific language) that generates an implementation of our domain contract. There’s a lot of meaty terms in that last sentence, let’s spend some time and break them down.

Common use case: compilers

First, I said that code generation is common, but behind the scenes. That’s because most compilers have some form of code generation happening in them. So for example, compiling Typescript for a Web app, or compiling Swift for an iOS app.

Let’s talk a little more about compilers. A compiler’s job is to take source code that humans can read and produce output that a computer can run. This is broken up into parts:

  1. Parsing the source code
  2. Stuff
  3. Generate output.

Parsing the source code is all about understanding text and turning it into a format a computer can do stuff with.

Stuff could be interesting things, like in Typescript, adding static type checking to vanilla Javascript. This usually involves manipulating an abstract syntax tree. For example, you could enforce rules in the tree and make sure everything is valid.

Generate output is the code generation part. The tree needs to turn back into something that the computer can run. For Typescript this is generating Javascript, for Swift, this is producing machine code.

A quick example

2 + 2

Given the above syntax, parsing turns it into a syntax tree:

Then for stuff maybe we want to enforce that a “Plus” always has “integers” connected to it. In this case, our tree looks good.

For generate output, we need a rule for what to do with a “Plus”. I’m making up this rule:

  • Load the left hand number into the computer.
  • Load the right hand number into the computer.
  • Add them.

So we apply that rule to our tree and get:

  • Load a 2 into the computer.
  • Load a 2 into the computer.
  • Add them.

Which is approximately what machine language looks like 🙂 I linked to NAND to Tetris which my friend Devin Mork showed me. It reminds me of my computer architecture course in college.

Domain Specific Languages

A domain specific language is making your own programming language. This lets you (1) create a language that is more meaningful and specific, and (2) produce generated code that does something useful.

One example is defining an API between two microservices. You could exchange URL’s back and forth between developers:

https://example.com/api/add-numbers
https://example.com/api/get-result

But then you’d also need to agree on an API body format (maybe JSON), whether things are a GET or a POST, and a myriad of other details.

What if instead you could use a DSL that let you define the following:

api Math {
    command add-numbers(
        leftHand: int,
        rightHand: int
    );
    query get-result(
    );
}

This defines something meaningful to the domain (we have an api with a command and a query), but hides the implementation.

The implementation could be a Web API, it could be a “REST API,” it could be something like Apple’s XPC. This implementation would be built out as a set of code generators, but it is not part of the domain definition. So for example, we could generate a Jersey controller:

@Path("/Math")
public class MathController {

    @GET
    @Path("/get-result")
    public Response getResult() {
        //something
    }

    @POST
    @Path("/add-numbers")
    public Response addNumbers(@QueryParam("leftHand") String leftHand, @QueryParam("rightHand") String rightHand) {
        //something
    }
}

This is a partial example, you’d need a bridge between the generated code and hand-written code. For example, maybe “something” is a call to an interface that a developer will implement by hand.

This is not a new idea, Martin Fowler wrote a book “Domain Specific Languages.” For web APIs see also grpc and api blueprint. If those meet your needs, I would use those instead of making the investment in building your own DSL and code generator.

Event sourcing and CQRS

As part of the #2018TechReadingChallenge, I’ve been working my way through Microservices Patterns, by Chris Richardson (currently a Manning MEAP).

Event sourcing and CQRS are two key architecture concepts in the book that have less to do with the “micro” in “microservices,” and more to do with software architecture in general. While they are often mentioned together, they are separate concepts, and solve slightly different problems.

Event sourcing fits in the general category of an append only model, which is a way of persisting the data in your application. Instead of storing the current state, it stores history that led up to the present. It stores these using events, which are immutable, and represent business intent. Events record a thing that has happened.

A related concept in domain driven design is an aggregate, which rebuilds state by querying the event history.

CQRS stands for Command Query Responsibility Segregation. This relates to an OO principle, command query separation, which classifies methods into “commands,” which mutate the data (but does not return data), and “queries,” which return data only. In CQRS, we extend this principle to the design of a subdomain, and separate the responsibility of writing the data from reading the data. The two sides often communicate using events.

It solves a “stale data” problem in a collaborative domain. A collaborative domain is a business domain where multiple users are working together on a set of data, and expect that data to be coherent. The data is only as stale as the lag between the command side and the query side. However, by separating these sides and putting a visible part of the architecture between them, we have raised awareness that there will always be stale data in a collaborative system.

Another advantage to CQRS is that the read side and the command side can be scaled independently, so for example, if the number of queries is vastly greater than the commands, you can add more nodes that handle the queries.

For more on these concepts, see the CQRS Journey page on Microsoft, specifically the Exploring CQRS and Event Sourcing e-book. I’m also drawing from Patterns for Building Distributed Systems for The Enterprise, by Michael Perry on Pluralsight.

Dreyfus model of skill acquisition

I just finished reading Pragmatic Thinking and Learning, by Andy Hunt, as part of my 2018 Tech Reading Challenge. (It fits in the category, “a book about an abstract concept.”) In lieu of a full review, I’m sharing some key takeaways from chapter 2, “Journey from Novice to Expert.”

The Dreyfus model of skill acquisition is a tool to help you understand  how you are doing in a new skill. It can apply to any skill area, such as cooking, writing, bowling, or software development. It comes from a paper by the Dreyfus brothers, published in 1980. In it, they identify five stages of skill development. I’ll stick these at the bottom of the article (It’s a bunch of bulleted lists) and talk about some of my takeaways.

Problems

The Dreyfus model helps us understand a few different common problems in the world of professional software development.

First is a tendency to over-rate our own abilities. This can happen when we don’t know what we don’t know. This is known as the Dunning-Kruger effect.

Second is when we decide to stop learning, and yet think that expert status has been achieved. This has been described as the expert beginner, and Erik Dietrich has a set of articles about it over at his blog.

The Dreyfus model does not address how large or small a skill is scoped, but it seems you can break this up into small pieces. For example, the skill of unit testing is separate from the skill of refactoring, or from designing for test. In The Art of Unit Testing, Roy Osherove lists these as separate skills.

No one size fits all

My big takeaway from the 5 stages is that it can help explain why I am uncomfortable in different problem solving situations. For example, if I’m being asked to fit into a strict set of rules, and yet solve a problem that is very familiar to me, I can chafe a little. The rules are more suited to a novice, and I’m feeling more confident. On the flip side, if I have junior developers on the team, it is a good thing to have conventions, standards, and some guidelines for them to follow. And if I’m pursuing a senior engineer role, it’s my job to create the guidelines.

Movements target certain skill levels

The different stages help us identify how movements can target different skill levels with different attributes. For example, Agile could be viewed as a set of rules, rituals, and ceremonies, which would appeal to the advanced beginner, or it could be viewed as a way to learn and correct past performance, which is something a proficient stage person can do.

A great example of this is applying design patterns. An advanced beginner sees the potential for patterns everywhere! And the end up with something like EnterpriseFizzBuzz.

Conclusion: Check out the book

I recommend picking up a copy of Pragmatic Thinking and Learning, by Andy Hunt. I didn’t even cover all of chapter 2—there’s a great case study on how the model was applied to the nursing profession, and the author ties this to software development. The rest of the book has some great tips for learning and problem solving. I highly recommend it.

The five stages

Novice

  • they need a recipe
  • don’t know what they don’t know
  • prefer walkthroughs, in person if possible

Advanced beginner

  • start to break away from the fixed rule set
  • learn new tasks as the need arises
  • can use advice in a similar context
  • want information fast, but not the big picture
  • no broad-based conceptual understanding of the task set
  • solo learning is more possible

Competent

  • Troubleshooting skills
  • “They can mentor the novices and don’t annoy the experts overly much”
  • informal leadership role
  • more deliberate planning
  • Still can’t apply agile methods to the full extent (!)

Proficient

  • can correct previous poor task performance
  • learn from the experience of others
  • apply maxims (Don’t Repeat Yourself)
  • know what they don’t know

Expert

  • work from intuition
  • unconscious competence (can’t explain why)
  • can take context into account
  • very rare!

Creating financial systems

Financial systems are “set it and forget it” applied to your personal finances. They are valuable because they save you from having to worry about your money. For example, instead of making a decision to save each month, it happens automatically. It helps you focus your attention on things that do matter, like creating value for other people.

My favorite guide on creating systems for personal finance is by Ramit Sethi, called The Ultimate Guide to Personal Finance.

In it, Ramit talks about a prescriptive budget and a descriptive budget. The descriptive budget is where your money is currently going. Due to my obsessive tracking of my expenses in Gnucash, I already had this. Do you need to be like me? Probably not—there exist tools like Mint.com that can aggregate various spending accounts into one convenient place.

A prescriptive budget is a plan for spending each month. Here’s one way to create a spending plan:

  • Fixed costs: things like rent, utilities, debt
  • Investments: your 401k and Roth IRA contributions
  • Savings: Build up an emergency fund if you haven’t already. Otherwise, this is for accumulation fund for things like vacation, house down payments, etc.
  • Guilt-free spending money: Groceries, eating out, shirts from REI, movies.

Then you want to create systems that automate the categories in your prescriptive budget. That way, you spend less time worrying about money and more time enjoying the guilt-free spending money.

Does that sound like more fun than constantly worrying about financies? If so, head on over to The Ultimate Guide to Personal Finance.

2018 Tech Reading Challenge

Many of us are challenging ourselves to grow in the new year. One area of growth I strive for is in my career as a software developer. To that end, I’m sharing my 2018 reading challenge. I hope you will find it stretches your technical ability.

How it works

  • Choose a reading goal early in the year and set your pace accordingly.
    • The beginner reading challenge is 1 book a quarter.
    • The advanced reading challenge is 1 book a month with 2 months off.
  • Choose the books and read them in any order, checking them off as you complete them.
  • Post about your progress on Twitter using the hashtag #2018TechReadingChallenge. (Cross-post on Slack too)

Ideas

  • Define your definition of “read,” since some technical books are pretty dry if you try to read them from cover to cover.
  • Ignore the categories and use #2018TechReadingChallenge about any software development book you read this year.

2018 Reading challenge (beginner—4 books)

  • A book about unit testing
  • A book about a framework, tool, or language
  • A book recommended by a coworker
  • A book published in 2018

2018 Reading challenge (advanced—10 books)

  • All the books in the beginner list plus:
  • A book about leadership
  • A book about an abstract concept
  • A book about software architecture
  • A book about agile
  • A book from the 80s
  • A book with an animal on the cover