John Fremlin's blog2018-04-29T00:00:00ZThe Rewrite Fallacy2018-04-29T00:00:00Z/rewrite-fallacy<div class='blog-entry-story'><p>Software developers love to criticise working code, say it is all very bad, and should be rewritten. It's generally possible to discourage the junior engineers who try this tactic by pointing out how big a project it would be. But some very experienced ones are looking for a big project, know that the rewrite might have few customer facing improvements, is likely to fail and come in over-budget and late, and still propose it. They claim that this time, it will be different. Why?</p><p>The upfront costs of understanding an old, working system are high and visible progress is low. The immediate visible productivity of a rewrite is high. There is the opportunity to output grand sounding designs and lots of code. Therefore the engineer will likely benefit from an unnecessary rewrite and appear productive doing it.</p><p>Unfortunately, the cost of understanding the problem domain will be borne. This will take time. Typically, there is a built in expectation that a project will be later than estimated &mdash; so paying this cost towards the end makes sense for the engineer.</p><p>The development time before the rewrite starts to benchmark itself against the prior system will look falsely productive. As soon as possible, the design and implementation of components of the new system should be validated against production. The new systems should immediately take over partial responsibility as they are built, or in some way have the workload is mirrored into them, to close this window of false productivity.</p><p>Will the rewrite be a net improvement? Unless the team is very familiar with the old system, there is little reason to believe this. In fact, as the old system is working and the new one is not, at least at the moment of deciding to fund the project, then objectively the new system must be worse to start with. Hopefully, it will improve to be better over time.</p><p>Hope is a huge benefit of a rewrite. Allowing a rewrite can greatly improve morale. The blessing of this optimism has to be balanced with the curse of optimistically ignoring problems.</p><p>Rewrites are controversial and often opposed by people with responsibility for outcomes because they are potentially a way for weak engineers to make a land grab and claim they did a good job with no accountability. Therefore engineers might be reasonably worried about revealing issues until so much cost has been sunk that issues will not be used to cancel the project. A typical tactic for this is to claim that nothing can be measured until the whole system is built and to refuse to make even small changes to the old system, which would greatly improve matters but make the rewrite less urgent.</p><p>In my experience, a small amount of imagination and a little work is enough to run in parallel with the old system. This work is likely to be resisted but it will reduce risk and most importantly will focus efforts and align people around tangible outcomes.</p><p>It's important to set the expectation that success will not be immediate, and balance the benefits of the hope with concrete measured progress. In the best case, the confidence and ownership gained from a rewrite can outweigh the costs &mdash; it's an investment in an engineering team's level of engagement. </p></div>How B players hire C players2018-03-19T00:00:00Z/how-b-players-hire-c-players<div class='blog-entry-story'><p>Jack Welch, GE CEO popularized segmenting workers into letter groups: A players (top 20%), B players (70%) and C players (bottom 10%). Steve Jobs is popularly quoted, <q>A players hire A players; B players hire C players; and C players hire D players. It doesn't take long to get to Z players. This trickle-down effect causes bozo explosions in companies.</q></p><p>I've been on interview loops for hundreds of candidates at companies that claim to want to hire the best. Out of those hundreds of interviews a few really outstanding candidates come up. We manage to get them rarely. If our aim is to hire the best, why do we often fail?</p><p>Often, an interviewer will have a negative reaction to the candidate, leaving feedback like <q>aced the interview but don't want to work with them.</q> This makes it difficult to justify a strong effort to hire. Once, I was on a loop with a superstar candidate. I and several others noted that. One of the other interviewers was a famous figure in an academic branch of engineering. The interviewee argued with the famous figure. It seemed to me that the candidate was probably right but the interviewer was so offended we couldn't extend an offer.</p><p>For nearly every firm, there are people who are much better and much worse for the job than those working there. The quote about A players hiring A players is justified with the idea that somehow people at the top won't mind hiring people better than them &mdash; as they might be more confident and willing to embrace challenging points of view. I don't think that's always true as the example above suggests.</p><p>A thought experiment that I've found useful is to imagine how a superstar candidate would experience the interview. I liked to bring this up when training interviewers. The questions would be super easy and interviewers themselves would make basic mistakes. How could such a candidate communicate in a way that did not appear arrogant?</p><p>Well I found out. I once interviewed such a senior candidate. I was asking a systems design question. The candidate would easily complete designs based on my suggestions but was cautious about suggesting anything himself. It was a little confusing as he was obviously very capable and knew how to design for all the different trade-offs I could think up. I felt I was missing something and gave him very positive feedback. The other feedback showed him acing the coding interviews. This superstar wasn't dinged for arrogance &mdash; he'd carefully avoided mentioning mistakes made by the interviewers. Unfortunately, he wasn't enthusiastic about joining our company &mdash; understandably. He would have been working with people much less capable.</p><p>This is a hard reality to reconcile with wanting to hire the best. People better than you might not want to work with you. Much more comfortable to hire those who do &mdash; but the benefits of a little humility can be extraordinary!</p></div>Nothing was a billion dollar mistake2018-01-20T00:00:00Z/billion-dollar-mistake<div class='blog-entry-story'><p>Tony Hoare, inventor of quicksort and many other foundational computer science concepts, claimed a <q>billion-dollar mistake. It was the invention of the null reference in 1965.</q> The special value of zero is used to indicate that a reference points nowhere. This is very efficient. But it's not a mistake. The mistake is to try to ban missing values. They often really are missing! </p><p>Java programmers particularly are afflicted by the NullPointerException, with millions of hits for this on Google. They abbreviate it to NPE, and bemoan it. They'll actually check whether they would throw it and then throw another exception instead &mdash; though Java code that distinguishes exception types is rare.</p><p>At a superficial level, the null pointer exception is the result of a mismatch of expectations: someone wrote some code that wanted a value, and someone called that code without giving it the value. Without looking deeper, an immediate obvious response is to systematically try to annotate or check that the values are really there when they are required.</p><p>The default behaviour in Objective C on SQL usually makes more sense to the output of the program, especially programs which should not crash, where null (or nil) just combines to form another null. Still, missing values disrupt the normal computation and thinking harder about what to do is often worth it.</p><p>There have been countless quixotic attempts to avoid this hard thinking: to ban null, to make complicated ways to declare that things are not nullable, and define away nullness. There is a pedantic tendency to try to harness punctuation elements like ? or define Option types &mdash; to try to force people to spend effort to admit that a value could be missing.</p><p>This is missing the point in the missing values: the hard problem isn't when someone is deliberately, maliciously trying to withhold information from the code they're calling. Instead, it's when they just don't have the information and are passing down what they have. The systematic response should generally not be to insist and demand the information. This would be convenient and make it very easy for one party in the exchange, as now we can write the code without figuring out what to do when there is incomplete information.</p><p>Unfortunately, incomplete information is the default state of the world. If the programmer won't deal with it, then the poor users of the program will have to, by entering bogus values. It's better to unambiguously indicate that something is missing rather than demand a lie!</p><p>Thousands of immigrants to the US have an invented first name, which is very inconvenient for them, because systems are set up to discourage a missing name: they are called Fnu. This is a bureaucratic consequence of a non-null annotation. It is used as an acronym for First Name Unknown, and in <a href="http://www.texasstandard.org/stories/why-fnu-is-the-official-first-name-of-many-afghan-allies-in-the-u-s/">a bizarre twist of fate, for people who only have a first name</a> &mdash; as their first name, so the last name is not null.</p><p>In the real world, incomplete information is the default. A convenient unambiguous representation is the null pointer, which doesn't take up space in terms of its binary representation or syntax &mdash; unlike the verbose imposition of Option types as in Haskell.</p><p>Incomplete information is the default with computers too. We need to learn to accept that. Recent language standardizations try to pretend it's not the case, forcing painful complexity on users.</p><p>The <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0088r2.html">C++ variant, a type-safe union</a> which was proposed in the last few years misses this lesson. In an opinionated attempt to ban null, there is no by default empty value for a variant. But a variant can be empty, they call it valueless_by_exception() and so code has to deal with it. This paradox afflicts non-nullable ! class fields in Kotlin too, which can be null before they're initialized. Requiring a null-check on something marked non-nullable is just silly.</p><p>Let's accept that information is incomplete, and be kinder when that's the case! </p></div>Freeing space from open files on UN*X when deleting doesn't work2017-12-30T00:00:00Z/truncate-deleted-files<div class='blog-entry-story'><p>Your disk is filling up, you run du | sort -n | less and identify the culprit, then rm -f. But df tells you that the disk space is still used.</p><p>According to <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/unlink.html">POSIX</a>, <q>the removal of the file contents shall be postponed until all references to the file are closed</q>. A program must have the file open. Obviously, an easy way out is to kill all the programs using it, but sometimes you can't kill the program because it's doing something useful. First find which it is with lsof -nP +L1.</p><p>That tells you the pid and fd numbers. The file could actually be opened in multiple ways! However, you only need to truncate it once - by running : | sudo tee /proc/$pid/fd/$fd for one of these pairs.</p><p>Note that if the file is mmap'd the process might get SIGBUS. If it's just a logfile, this is unlikely!</p><p>Some people choose to first truncate before deleting logfiles to avoid chasing through this, i.e. : &gt; $file; rm $file. </p></div>LISA17 conference notes2017-11-02T00:00:00Z/lisa17-conference<div class='blog-entry-story'><p>I attended the LISA17 conference in San Francisco. Here are some very rough and opinionated notes from talks that interested me. If I made any mistakes let me know and please comment on the things I missed!</p><p><h3>Ted Ts'o: <a href="https://www.usenix.org/conference/lisa17/training-program/session/linux-performance-tuning">Linux performance tuning</a></h3></p><p>This tutorial was exceptional because the speaker has years of experience as one of the top Linux developers. Ted uses emacs!</p><p>Goals of performance tuning <ul> <li>improve throughput</li> <li>manage and understand performance cliff</li> </ul></p><p>Investigate bottlenecks with rigorous scientific method: change one parameter at a time. Take lots of notes and make haste slowly. Measurement tools can impact behaviour of the observed application.</p><p>Basic tools to start with <ul> <li>Memory: free -m. Adding memory might be the easiest way to speed up the machine.</li> <li>Tasks: top. What are the CPUs doing or waiting for?</li> <li>IO: iostat -x -k 5. This shows average queue size (avgqu-sz) for requests and merging statistics - crucial for understanding performance given larger requests are much more efficient. </li> </ul></p><p>Example filesystem benchmark: <a href="https://github.com/josefbacik/fs_mark">fs_mark</a> -s 10240 -n 1000 -d /mnt - creates 1000 files each 10kB in /mnt and does fsync after each. Can be greatly improved by relaxing sync guarantees. Use the exact benchmark for your application!</p><p>Cache flush commands are <q>ridiculously expensive</q>. Google runs in production with journal disabled, as it is so much faster and there is a higher level consistency guaranteed by multi-machine replication. This cross-machine cluster file-system also means RAID is unnecessary.</p><p>Ted Ts'o made this <a href="https://sites.google.com/site/linuxperftuning/snap">snap script</a> for low impact performance monitoring extraction from production systems.</p><p><h4>Storage</h4></p><p>In terms of selecting hardware: note seek time is complicated and should typically be reported statistically - worst case and average. Low number offsets LBA at the outer diameter of the disk can be much faster to seek. Therefore you can put your most used partitions at the first offsets of the disk to get a lot more performance - this is called <q>short-stroking</q> and can be a cheap way to get more performance. Filesystem software moves slowly as it has to be reliable and hardware generally moves much faster.</p><p>HDDs at 15000rpm run at hot temperatures and use a lot of power; many applications that used those have moved to SSDs. SSDs can also use a lot of power. They tend to fail on write. Random writes can be very slow - 0.5s average, 2s worst case for 4k random writes. You can wear out an SSD by writing a lot. See <a href="https://research.google.com/pubs/pub44830.html">Disks for Data Centers</a> in terms of advice about selecting hardware (Ted wrote this). Think about how to use iops across the fleet (hot and cold storage). The interface SATA 1.5Gbps or 3Gbps or PCIe may not be important given that e.g. random writes are slow. <q>RAID does not make sense</q> generally in today's world (at Google scale) and can suffer from read/modify/write cycles.</p><p>We can think about application specific file-systems, now we have containers. For example, ReiserFS is good for small files, XFS good for big RAID arrays and large files. Ext4 is not optimized for RAID.</p><p>Consider increasing journal size for small writes. Note Google disables the journal altogether.</p><p>Recommends <a href="https://github.com/brendangregg/perf-tools">Brendan Gregg's perf tools</a> using ftrace. These were introduced at <a href="https://www.usenix.org/conference/lisa14/conference-program/presentation/gregg">LISA 14</a> <ul> <li>iosnoop - friendly blktrace</li> <li>bitesize - distribution of IO sizes</li> <li>opensnoop - file opens</li> <li>iolatency</li> </ul> Also more advanced versions with lower overhead due to computing aggregates in kernel using eBPF, <a href="https://github.com/iovisor/bcc">the BPF Compiler Collection (BCC)</a>: biosnoop, bitesize, biolatency, opensnoop.</p><p>Consider the multiqueue scheduler for very fast storage devices like NVMe.</p><p><h4>Network tuning</h4> Immediately check for trivial basic health: ethtool, ifconfig, ping, nttcp. Check for various off-load functions and that the advanced capabilities of the card are used.</p><p>Consider whether you want latency or throughput. Optimize the bandwidth delay product. Then remember that increasing window size takes memory; this can be tuned with net.core.rmem_max and net.core.wmem_max. Use nttcp to reduce buffer sizes as much as possible to avoid bufferbloat.</p><p>UDP might be a better bet.</p><p>However, we can push TCP to be low latency. Disable Nagle with setsockopt TCP_NODELAY. Enable TCP_QUICKACK to disable delayed acknowledgments.</p><p><h4>NFS performance tuning</h4></p><p>Recommends considering an NFS appliance, e.g. NetApp.</p><p>Some tricks: use no_subtree_check. Bump up nfs threads to 128. Try to separate network and disk IO to different PCI buses - no longer necessarily relevant with PCIe. Make sure to use NFSv3 and increase rsize/wsize. Mount options: rw,intr. Make sure to tune for throughput, large mtu and jumbo frames.</p><p>NFSv4: only use NFSv4.1 on Linux 4+, see <a href="http://filesystems.org/docs/nfs4perf/nfs4perf-login.pdf">;login magazine, June 2015</a>.</p><p> <h4>Memory tuning</h4></p><p>If there is any swapping, first, try adding more memory. Add more and faster swap devices.</p><p>Paging will happen. majflts/s is rate of faults that result in IO. pgsteal/s is rate of recycling of page cache pages.</p><p>Try sar -W 3 and periodically send sysreq-m.</p><p>Note the memory hierarchy is important as closer caches are much faster.</p><p>Translation Lookaside Buffer (TLB) caches translation from virtual address to physical address. Can avoid up to six layers of lookup on 64-bit system - costing thousands of cycles. There are only 32 or 64 entries in the TLB in a modern system.</p><p>The jiffies rate can greatly affect TLB thrashing by controlling rate of task switches. Hugepages avoid consuming these entries. Kernel modules burn TLB entries while the originally loaded kernel does not.</p><p>The perf tool can show TLB and cache statistics. </p><p><h4>Application tuning</h4></p><p>Experimentation with eBPF.</p><p>For JVM consider GC and JIT. Size the heap.</p><p>Tools: strace, ltrace, valgrind, gprof, oprofile, perf (like truss, ktruss). Purify might not be as good as valgrind.</p><p><q>perf is the new hotness</q>. Minimal overhead, should be safe in production from a performance perspective. However, the advanced introspection capabilities may be undesirable for security.</p><p>There are many interesting extensions - like the <a href="https://github.com/iovisor/bcc/blob/master/tools/ext4slower_example.txt">ext4slower program</a> which shows all operations on ext4 that take longer than 10ms.</p><p>Userspace locking: make sure to use futex(2).</p><p>Consider CPU affinity.</p><p><a href="http://norvig.com/21-days.html#answers">Latency numbers that all programmers should know.</a> Note this does not include random write for an SSD because that depends on a great many factors.</p><p><h4>Conclusion</h4></p><p><q>It's more addictive than pistachios!</q></p><p><q>It's time to shoot the engineer and put the darn thing into production.</q></p><p>Great way of learning about the whole stack!</p><p> <h3>Robert Ballance: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/ballance">Automating System Data Analysis Using R</a></h3></p><p>This talk presented a valuable philosophy and attitude: that we should consider making repeatable re-usable reports. This goes against the grain of expectations around reporting which often frame reports as one-off tasks. The examples were very compelling.</p><p>Some background: R was written at Bell Labs by statisticians who were very familiar with UN*X. Data is dirty. <Q>The computations and software for data analysis should be trustworthy: they should do what they claim, and be seen to do so.</Q></p><p><q>I've spent my entire career getting rid of spreadsheets.</q> </p><p>Very rapid growth in CRAN R packages. Pipe operator %&gt;%.</p><p>Used dplyr. Small repeatable pipelines for reports that can be reused. Very pretty code examples using dplyr and ggplot and the aforementioned pipe operators.</p><p><h3>Renee Lung: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/lung">Testing Before You Scale & Making Friends While You Do It</a></h3></p><p>Your customers shouldn’t find problems before you do.</p><p>Onboarding a big new account with an expected 20k incidents per day, around 7M per year.</p><p>They wanted to test the load. <q>The only thing that behaves like prod, is prod.</q></p><p>Chaos Engineering is about experiments in realistic conditions. PagerDuty has Failure Friday - where they expose systems to controlled experiments.</p><p>Balance business and engineering.</p><p>Decided to create a tool to simulate load.</p><p>Noticeable customer impact from first and second test but they still persisted which was quite brave. The talk was very honest about the interpersonal and organisational issues that the project faced.</p><p>Tried to explain why the staging environment is different from production to an idealistic questioner.</p><p><h3>Baron Schwartz: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/schwartz">Scalability Is Quantifiable: The Universal Scalability Law</a></h3></p><p>Eben Freeman's talk on queuing is really good!</p><p>Recommends a <a href="https://www.youtube.com/watch?v=PGLYEDpNu60">talk by Rasmussen</a> on failure boundaries. </p><p>Failure boundary is nonlinear.</p><p>Hard to apply queuing theory to the real world of capacity and ops, as difficult to figure out how much time is spent queuing in real systems.</p><p>Add a crosstalk (coherence) penalty with a coefficient k as a quadratic term to the denominator in Amdahl's law. The penalty represents the cost of communication.</p><p>Defines load as concurrency.</p><p>Suggests that load-tests should try to fit the crosstalk penalty and Amdahl's law parameters. Claims that this fits quite well to many real world scaling problems with some abstract examples.</p><p><h3>Chastity Blackwell: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/blackwell"> The 7 Deadly Sins of Documentation</a></h3></p><p>Without effort, documentation will be scattered across multiple systems and notes that the costs are paid in ramping up new people. We should invest in documentation.</p><p><h3>Blake Bisset; Jonah Horowitz: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/bisset">Persistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful SRE Program Like Netflix and Google</a></h3></p><p>SRE is not a rebranded ops, should not try to build an NOC.</p><p><h3>Sasha Goldshtein: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/goldshtein">Fast and Safe Production Monitoring of JVM Applications with BPF Magic</a></h3></p><p>Beyond performance, we can trace things like system calls to find out where something is happening - for example, the stacktrace of the code that is printing a message.</p><p>The JVM can cooperate by adding more tracepoints -XX:+ExtendedDTraceProbes.</p><p>The advantage of BPF as opposed to perf, is that BPF can filter and aggregate events in the kernel, which can make things much faster than perf, which just transmits events. BPF can calculate histograms and so on.</p><p>Needs recent kernels - 4.9 kernel for the perf_events attaching.</p><p><h4>DB performance tracing</h4></p><p>Many performance investigations can occur now without modifying any programs. For example, there are instrumentation scripts like dbslower that can print out which queries in MySQL and can be extended to other databases.</p><p>We can trace and find out the exact stacktrace where a query is printed.</p><p><h4>GC performance tracing</h4></p><p>Can trace GC: ustat tool and object creation with uobjnew.</p><p><h4>Trace open file failures</h4></p><p>Use opensnoop to find failed open syscalls. Then attach a trace for that specific error to a Java application.</p><p><h3>Michael Jennings: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/jennings">Charliecloud: Unprivileged Containers for User-Defined Software Stacks in HPC</a></h3></p><p>Want to make it possible for people to bring their own software stack to run on the supercomputers at Los Alamos, and decided to explore containers. Unlike virtual machines, they do not have performance impact on storage or networking (Infiniband).</p><p>Recommends this LWN article: <a href="https://lwn.net/Articles/531114/">Namespaces in operation</a>.</p><p>Docker with OverylayFS can be slow on HPC. Therefore they built a system called Charliecloud with minimal performance impact, and native file-system performance.</p><p><h3>Matt Cutts and Raquel Romano: <a href="https://www.usenix.org/conference/lisa17/presentation/cutts">Stories from the Trenches of Government Technology</a></h3></p><p>Sometimes don't have source code or access to logs.</p><p>Many basic problems: 5% of veterans incorrectly denied healthcare benefits from a single simple bug.</p><p>Great value delivered by bug bounty programs.</p><p>API first!</p><p>Meaningful contribution - hugely impactful, bipartisan problems. Looking for software engineers and site reliability engineers for short tours of duty.</p><p><h3>Jake Pittis: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/pittis">Resiliency Testing with Toxiproxy</a></h3></p><p>CEO at Shopify once turned off a server as a manual chaos monkey.</p><p>Continued to work on resiliency, with gamedays. Then thought about automating the game days to ensure that issues remain fixed and don't regress.</p><p>Want to maintain authenticity.</p><p>ToxiProxy interposes latency, blackholing, and rejecting connections in the production systems and then is supported by automated testing in Ruby that asserts behaviour about the system.</p><p>Incident post-mortem fixes are checked and verified by injecting the faults again and checking application specific consequences. This confirms that fixes worked, and continue to work in the future.</p><p>Resiliency Matrix declares expected dependency between runtime systems. ToxiProxy tests allow one to validate that the dependency matrix truely reflects the production reality.</p><p><h3>Brendan Gregg: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/gregg">Linux Container Performance Analysis</a></h3></p><p>Common system performance tools like perf do not work well in containers, as the pid and filesystem namespaces are different. System wide statistics (e.g. for free memory) are published to containers which causes programs to make wrong assumptions: for example, Java does not understand how much memory is actually available in the container.</p><p>The situation is improving and there is ongoing integration of support for monitoring performance of containerized applications.</p><p>Understanding which limit is hit in the case of CPU containerization can be very confusing as there are many different limits.</p><p>PS. Brendan's talk from last year at LISA16 gives a general introduction to the advanced bpf tools: <a href="https://www.youtube.com/watch?v=UmOU3I36T2U">LISA16: Linux 4.X Tracing Tools: Using BPF Superpowers</a></p><p><h3> Teddy Reed and Mitchell Grenier: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/reed">osquery—Windows, macOS, Linux Monitoring and Intrusion Detection</a></h3></p><p>Labs showing how to collect and query many system level properties like running processes from a distributed set of systems with a tool called osquery.</p><p>It can collect current state and also buffer logs of changes.</p><p><h3>Heather Osborn: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/osborn">Vax to K8s: Ticketmaster's Transformation to Cloud Native Devops</a></h3></p><p><a href="https://tech.ticketmaster.com/2016/11/08/getting-to-innovation-faster/">Tech Maturity model</a>.</p><p>20k on-prem VMs.</p><p> <h3>Kevin Barron: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/barron">Coherent Communications—What We Can Learn from Theoretical Physics</a></h3></p><p>Human communications take a lot of time and we need to be careful that we're really communicating.</p><p><h3>Evan Gilman and Doug Barth: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/gilman">Clarifying Zero Trust: The Model, the Philosophy, the Ethos</a></h3></p><p>Establish some strong properties: that all flows are authenticated and encrypted.</p><p>No trust in the network. Automation based policy based on a Ruby DSL and Chef that reconfigures iptables rules to add IPSec routes between application tiers.</p><p>Related to Google's <a href="https://research.google.com/pubs/pub43231.html">BeyondCorp</a>.</p><p>Beyond the control aspects, another value of the approach is observability. Mentioned that another way of doing this is <a href="https://www.envoyproxy.io/">Lyft Envoy</a>.</p><p>Mostly build your own still.</p><p><h3>Brian Pitts: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/pitts"> Capacity and Stability Patterns</a> </h3></p><p>Very thoughtful talk with a comprehensive coverage of various techniques.</p><p>EventBrite has 150M ticket sales per year. Very spiky traffic. Over one minute can quadruple.</p><p>Bulkheads: partition systems to prevent cascading failures.</p><p>Canary testing: gradual rollout of new applications.</p><p>Graceful degradation.</p><p>Rate limiting. Understand capacity and control amount of work you accept.</p><p>Timeouts. Even have to timeout payment processors.</p><p>Caching.</p><p>Capacity planning.</p><p><h3>Corey Quinn: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/quinn">"Don't You Know Who I Am?!" The Danger of Celebrity in Tech</a></h3></p><p>High energy and well presented talk.</p><p>Netflix: developers have root in production.</p><p>Should not be cargo-culted to places without same culture of trust and top quality talent.</p><p> <q>Be careful about punching down.</q> Recognise the weight that your words carry coming from a successful company with specific constraints.</p><p>Culture of security conferences is toxic.</p><p> <h3>Ben Hartshorne: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/hartshorne">Sample Your Traffic but Keep the Good Stuff!</a></h3></p><p>Adapt sample rate as you're watching your traffic, to scale observability infrastructure logarithmically with production. Sample rate should be recorded in event, and reduced in proportion to traffic.</p><p>Honeycomb does this with <a href="https://github.com/honeycombio/honeytail">honeytail</a>. Another alternative is Uber's opentracing: <a href="https://uber.github.io/jaeger/">Jaeger</a> which uses a consistent sampler.</p><p><h3>Mohit Suley: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/suley">Debugging @ Scale</a> </h3></p><p>Distributed tracing is the new <q>debugger</q>.</p><p>Use Twitter's <a href="https://github.com/twitter/AnomalyDetection">anomaly detection</a> R library.</p><p><h3>Jon Kuroda: <a href="https://www.usenix.org/conference/lisa17/conference-program/presentation/kuroda">System Crash, Plane Crash: Lessons from Commercial Aviation and Other Engineering Fields</a></h3></p><p>Need to better at following checklists, sterile cockpit rule (kicking out unqualified people). Avoid normalization of deviance. Lots to learn from airline industry!</p><p>Think about telemetry.</p></div>Square CTF 2017 Grace Hopper2017-10-19T00:00:00Z/square-ctf-2017<div class='blog-entry-story'><p>Square put on <a href="https://squarectf.com/">a great competition this year</a> at the Grace Hopper conference. My girlfriend was attending and had solved a lot of the challenges but some of the high pointers were left.</p><p><h3>6yte</h3></p><p>The <a href="https://squarectf.com/challenges/6yte">6yte</a> challenge hooked me. The task was to exploit an x86 32-bit Linux binary to print a file to the console - in only 6-bytes of machine code. Most opcodes on x86 are at least two bytes, so this means using at most three instructions. A tight budget!</p><p>The 6yte program was pretty simple. It memory mapped an executable region then decoded its command line argument as a hex string into this region, then jumped to it. It also printed out the addresses the program was loaded in.</p><p>On one side, this is much easier than exploiting a typical application nowadays, which probably marks writable memory non-executable. On the other hand, the 6yte program calls alarm() so that if you pause in the debugger, it will just crash, and it also uses sleep() to delay, so you can't immediately just try tons of things at random. These countermeasures made the contest much more fun for me.</p><p>I spent quite a while being misled by the printing of the program addresses into thinking I should use that. I wanted to call the puts function that is used elsewhere in the program to print out the string. In the process I learnt a lot about the <a href="https://systemoverlord.com/2017/03/19/got-and-plt-for-pwning.html">Procedure Load Table</a>. Trying to compute the relative address and then generate the right relative jump confused me. My plan was to spend one byte pushing edi onto the stack, and then five bytes jumping to puts(), or try to push the address of puts() onto the stack then jump to it, or something along those lines, but I just couldn't squeeze it into the remaining five bytes. Time to look more closely at the problem!</p><p>The disassembly for the jump to the decoded shellcode first loaded the target string into eax and then put it in edi <pre> 0x80488d2: 8d 85 68 ff ff ff lea eax, dword [ ebp +0xffffff68 ] 0x80488d8: 89 c7 mov edi, eax </pre></p><p>Then we were given some handy register values. Comparing to <a href="http://asm.sourceforge.net/syscall.html#p32">Linux system call numbers</a>, 4 is very helpful because it means write, and STDOUT_FILENO is 1. We are all set up to make a syscall!</p><p><pre> 0x80488da: ba 05 00 00 00 mov edx, 0x5 0x80488df: bb 01 00 00 00 mov ebx, 0x1 0x80488e4: b8 04 00 00 00 mov eax, 0x4 0x80488e9: ff 65 e8 jmp dword [ ebp + 0xffffffe8 ] </pre></p><p>To execute the system call we need just two bytes cd80, but first we need to mov edi to ecx (with 89f9). This will unfortunately only print the first 5 bytes, as defined in edx, but we have two bytes of instructions left. I tried several ideas for increasing edx, like adding it to itself it and shifting it and so on, but then remembered the wonderful lea instruction on x86. This oddly named instruction doesn't actually access memory. It combines a mov, and a shift add - a limited multiply accumulate.</p><p>To find out opcodes, I was grepping objdump --source /lib32/libc-2.24.as that has a good sample of opcodes. A shortcut to avoid having to run an assembler. I discovered that ecx <- edi + <i>xx</i> is 8d4f05<i>xx</i>. This costs four bytes, and then the last two can be used to do the int 0x80 syscall. Not the neatest solution (<a href="https://medium.com/@c0ngwang/square-ctf-writeup-6yte-c3f02cbb919b">Cong Wang has a much better one</a>) but it let me read out the flag :)</p><p>By now I was pretty enthused with the competition - it was my first time crafting an exploit!</p><p><h3>Needle in the Haystack</h3></p><p>The next problem I tried was the <a href="https://squarectf.com/challenges/needle-in-the-haystack">Needle in the Haystack</a> forensics challenge. Many years ago I implemented a vfat filesystem and fsck program, so I was very happy to see that it had a FAT12 filesystem. Inside was a Ruby on Rails project. There were some developer passwords in it which I immediately put into the submission box for the challenge - and was punished with a <a href="https://www.youtube.com/watch?v=dQw4w9WgXcQ">RickRoll</a>. That teasing riled me and though it was late on Thursday night after my floundering around on the previous problem, my motivation redoubled.</p><p>It took me a while to realise that there was nothing in the Ruby on Rails project. I compared it against the base skeleton created by the Rails setup by default. This was my first CTF, I didn't know the conventions. I wasn't sure how much work was expected and what to focus on.</p><p>I tried a bunch of data recovery programs to dig out deleted files from the filesystem image, and found a tarball containing a git repository. I checked out all revisions in all branches, some of which were very temptingly labeled, but there didn't seem to be anything in it, so I tried a bunch more undeletion programs, and then switched to solve smaller challenges.</p><p>This gave me more idea of the conventions used in the competition. Reading the rubric for the haystack challenge I noticed it mentioned the word blob: that meant the git repo was the right track: and git fsck --lost-found gave up the answer.</p><p>This ended up being my favourite problem because it combined so many technologies (Ruby on Rails, FAT, git) and tantalized with apparent victory at multiple steps.</p><p><h3>Competition!</h3></p><p>Other questions were also fun. I really enjoyed the SQL challenge - my first time doing a SQL injection, for which the SQLite function GROUP_CONCAT was very helpful. Then I found out that there was an easier version of 6yte without the bytes limit. Dammit!!</p><p>Now it was super late at night but team chocolates was #2 on the leaderboard. The challenges had all been so much fun and the VMs and setup for each were super well done, so I was very enthusiastic and feeling very hungry for the #1 spot. The next morning I was very excited as I thought the competition would end on that Friday (Alok eventually explained to me that we had an extra week) and I ran round to round up people to help. I didn't want to lose and was wondering about getting IDA Pro to solve the floppy challenge. Turns out Zach and <a href="https://trmm.net/GHCCTF">Trammell</a> were the right choices and chocolates got to #1 while I slept.</p><p><h3>Thanks</h3></p><p>The contest was a ton of fun. It was my first time attempting shellcode and first time doing SQL injection. It makes me really appreciate the value of having a library of tools in security research. The field has developed its own culture, experts and can seem impenetrable (pun!). This CTF made it accessible to me in a way I never anticipated.</p><p>I learnt new techniques: about instruction coding, dynamic linking, and about process from Trammell. He showed me how to connect qemu in debug mode and <a href="https://www.hopperapp.com/">Hopper</a>. The way he approached the problem was very different to my unskilled initial expectations: I thought of trying to extract the logic into a sandbox and would have been tripped up on all the countermeasures in the code, whereas Trammell's confident holistic reverse engineering quickly neutralised them.</p><p>In terms of the mechanics, the contest framework, with VMs being spun up, was ambitious and worked perfectly. On a non-technical level, the jokes and teasing with RickRolls and countermeasures made the challenges personal. Solving the problems was joyous. It left me very impressed with the culture at Square, that visibly invested so much. The contest was run really well with great responsiveness on Slack, and I'd love to do more CTFs. Thanks Square! </p></div>Zero value abstractions2017-10-07T00:00:00Z/zero-value-abstractions<div class='blog-entry-story'><p>The Rust and C++ communities have embraced the idea of abstractions without runtime overhead. Object orientated programming encourages the idea of dynamic dispatch - at run-time choosing what to do based on the type. This is costly: a small cost as a decision has to be made at runtime and a potentially very expensive consequence: the compiler can't inline and propagate constants. However, it allows code to be written once which works with many types. So called zero cost abstractions avoid this by having the compiler figure out the specific concrete implementation behind the abstraction and then perform its optimizations with this information.</p><p>Runtime cost is actually only part of the cost of an abstraction. Even if there is no runtime cost, abstractions must provide value as they have other costs. An abstraction introduces a new concept and imposes a mental burden on the people working with it. In the ideal case, the abstraction is perfectly aligned with the problem domain: for example, it's often very convenient to be able to show something on the screen and get its dimensions independently of whether it is a photo or a video &mdash; abstracting over the difference reduces the amount of code written, and makes it clear that the code doesn't care about those details. This may actually be good for people debugging and reading the code.</p><p>Abstractions defined in the wrong way can make it hard to modify code: by muddling together unrelated things, by hiding useful details, increasing compile times, or just by confusing people and taking up mental space. These costs are less easy to measure than the runtime cost. However, they can be much more expensive. Debugging code from a stagnant project, where the build environment isn't readily available, is vastly harder when there are layers of abstraction. Abstractions obscure the answer to the question: what does this code actually do?</p><p>Weak engineers can try to <q>abstract</q> away the parts of the project they don't know how to accomplish. No value is being added there. Another abuse is in wrapping existing code and interfaces belonging to another project or group: this sort of wrapping layer is very easy to write and gives an illusion of productivity, but means that the people who own the code will now have to understand a wrapper in order to help.</p><p>It's fun to reduce runtime costs. However, given the other costs are normally more significant, it's important to think of the value that the abstraction brings. An abstraction needs to be valuable even if there are no runtime costs. How much does it really help?</p><p>The worst abstractions abstract nothing, and provide no value: most commonly, a grand interface with a single implementation. They impose a cost on all readers &mdash; slogging through meaningless code, and slow people debugging production issues, who eventually have to understand that the interface is a mask for the real underlying implementation. Abstractions are costly. When reviewing or writing code, remember abstractions must provide value. </p></div>Pagination, a great software interview question2017-05-06T00:00:00Z/pagination-question<div class='blog-entry-story'><p>Experienced, highly productive, senior engineers at companies like Google have told me they take weeks to prepare and practice for interviews as they don't use those skills in their jobs. <A href="https://mobile.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html">Brainteasers are a complete waste of time</a> but it's hard to find questions which are relevant to the job without being overly specific.</p><p>However, pagination is a great question: from ranked search to simply retrieving a list from a database, cross-machine APIs are very inefficient if they try to gather and send everything at once or just one thing at a time; instead they should fetch items in batches (an obvious but impractical extension of this idea is the <a href="https://en.wikipedia.org/wiki/Lambda_architecture">lambda architecture</a>). Pagination is great as an interview question because it is clearly related to the job, trips up software engineers regularly, and can be asked in calibrated stages. A huge amount of software engineering involves cross machine APIs nowadays, so the question is not specific to one area.</p><p>1. Screener question: API for fetching in pages of a given size from a fixed list from a single server. If the candidate can't do this then he or she will almost certainly struggle to implement any sort of API. Pivot to ask another simple question (like: intersect two lists of personal contact information), to assess whether the candidate was just confused by the framing.</p><p>2. Main question: API for fetching pages in a given size for a list from a single server where items can be deleted or inserted while the fetch is happening. It's helpful to give some business context: an extreme case is a list of messages in discussion forum app that supports deletes. This forces the solution away from the obvious idea of keeping a consistent snapshot of items for each client on the server. The client has state from previous pages that it can send to the server.</p><p>Once the candidate can solve the problem even if things are deleted or inserted at inconvenient times, to assess the quality of the solution: ask how much information needs to be communicated each fetch? Trivially, the client could send back everything it knows but that destroys the benefit of batching. Ideally, the client would send back a fixed size cursor. Secondly, how expensive is it for the server to compute the next page?</p><p>Some candidates get frustrated and try to demand that the DB, IPC or API framework provide this consistent paging. That would indeed be wonderfully convenient but would imply complex integration between the datastore and the IPC layer &mdash; and the applications specific tradeoffs around showing stale data. Concretely, consistent paging is not offered by popular frameworks for these reasons.</p><p>3. Advanced: ranked distributed results. Many systems are too big to have the list of items stored in a single server. For example, Internet search nowadays involves interrogating many distributed processes &mdash; more prosaically a hotel booking website might ask for inventory from many providers. Rather than waiting for all to respond the client should be updated with the information at hand. Items can suddenly be discovered that should be at the top of the list, how should that be handled? Larger scale examples demand a co-ordination process on the server side that aggregates and then sends the best results to the client. The extent of co-operation with knowledge of client's state depends on the context. How should resources be allocated to this coordination process and how can it be timed out?</p><p>The question provides good leveling because it is implicitly required for many straightforward tasks (like providing a scrollable list in an app) but then is a key outward facing behaviour of large systems. In terms of computer engineering it touches on a range of knowledge in a practical setting: data-structures and algorithms to coordinate state. The client effectively has a partial cache of the results, and caching is known to be hard. Finally, the extension to distributed ranking should spur a mature discussion of tradeoffs for more sophisticated engineers. </p></div>Kotlin is a better Java2017-03-13T00:00:00Z/kotlin-is-a-better-java<div class='blog-entry-story'><p>The <a href="https://kotlinlang.org/">Kotlin</a> programming language has all the features of Java, and supports all sorts of helpful things like extension functions, stackless inlines and named parameters. If you're starting a new project for the JVM, shouldn't you just use Kotlin?</p><p>In 2010, I asked the <a href="scala-is-not-a-better-java">same question about Scala</a> &mdash; the answer was no. Scala aims for a target that is not at all the practical minimalist Java. Instead, it's a hodgepodge of half-implemented ideas from C++ and Haskell. Astonishingly, a <a href="https://issues.scala-lang.org/browse/SI-1338">basic for loop in Scala is translated into a bytecode litterbug</a> that spews garbage every iteration. Kotlin, on the other hand, keeps it clean and even makes it easy to invent new efficient iteration styles with guaranteed stackless inlining of lambdas (when annotated).</p><p>The features of Kotlin improve the life of a Java programmer. The language doesn't indulge in whimsical flights of fancy, like attempting to redefine what nullness is by inventing an Option datatype. The JVM has an efficient representation for a missing value: null. The only problem is that the Java language designers decided to crash the program by throwing a NullPointerException as the default response to this common condition. Objective C is much more sensible and <a href="http://nshipster.com/nil/">just ignores operations on the nil object</a> (though it does crash on NULL). Kotlin introduces syntax to keep track of what can be null, offers Objective C like behaviour with the ?. operator and provides :? to turn null into a default value &mdash; the Elvis operator. All in all, an elegant series of responses to <a href="http://lambda-the-ultimate.org/node/3186">the billion dollar mistake</a>.</p><p>There are areas where Kotlin violates the Java principle that everything should be explicit: the most significant is extension methods, i.e. syntactic sugar to allow extra methods for other people's classes, and second with var (and val) to declare variables without repeating their type &mdash; like C++'s auto, Go's var, etc. Both features have been used successfully in C# for many years. In terms of understanding a program, these violations are less severe than defaulting all functions to be virtual &mdash; which Java started with &mdash; allowing child classes to confusingly modify their parent's intentions.</p><p>The Android Kotlin extension methods remove so much boilerplate and avoid the kind of nasty surprise that Scala introduces (the Kotlin runtime is under a megabyte). In the end, they make the intention of the programmer much clearer by skipping the ceremony imposed by badly designed APIs.</p><p>Java is deliberately not programmer-orientated. That's the point of using it &mdash; it was designed to restrict the kind of trouble programmers can get themselves into. If you're stuck with the JVM &mdash; and as more than 80% of smartphones run Android, we all are &mdash; the extra help that Kotlin affords in terms of null-checking and syntactic sugar actually make programs clearer and safer. The complicated excesses and action at a distance of C++ or Ruby are avoided. Admittedly, you can't write programs so succinctly as Ruby or with anything close to C++'s performance, but the bar we are evaluating is Java. Is Kotlin a better Java?</p><p>Yes. Yes, you should use Kotlin despite it being new, and despite the consequent teething problems (<a href="https://youtrack.jetbrains.com/issue/KT-13275">an example</a>, promptly bug-fixed). The pace of improvement, incredible integration with IntelliJ (Android Studio) and great pragmatic design make it a winner: Kotlin swiftly (pun intended) to the rescue! </p></div>Recruiting software engineers and their CVs2017-03-04T00:00:00Z/recruiting-software-engineer-cv<div class='blog-entry-story'><p>Having conducted hundreds of software and other interviews and trained many interviewers, I've seen a ton of CVs. The one thing that will more or less guarantee a good interview performance is a strong TopCoder record.</p><p>The ability to solve algorithm puzzles under stress and time pressure is exactly what the coding interviews are about, and TopCoder tests these abilities at the highest levels. After some point in the rankings, it isn't just into people who can think fast and solve puzzles. The best players train regularly and continually improve, and in fact have to put in incredible discipline to outperform very dedicated opponents. These engineers in the end have the staying power to solve very complex system problems and the flexibility to attack them with novel approaches where necessary.</p><p>Only a small number of people can be the best at TopCoder. A wider pool of engineers can do a good job. Did they do a good job in the past? Good CVs boast quantitative production or business metric improvements. Bad ones describe techniques applied.</p><p>Experience isn't equally granted over time: for example, an engineer can work for years on an implementation that never goes to production and not really learn anything and just get set in his or her ways. The more feedback an engineer receives, and learns from, the more experience they get. People who have taken charge and launched an app or maintained a popular open source project in their spare time might have more real technical experience than a tech lead at a big company.</p></div>