John Fremlin's blog

The bonds of SQL

Posted 2015-06-17 05:45:40 GMT

Question for a SQL test: write a query to return the top five sales for each day in the database? It's easy to express this query for a given day and many databases have extensions for writing this query - but it can't be expressed portably in standard SQL. And this query falls squarely into the core use-case that SQL is touted to solve.

The No-SQL key-value store movement exemplified by databases like MongoDB is often lambasted for ignoring the lessons of history. SQL, a venerable ANSI standard, represents that history and provides a well known language and a protocol to more or less decouple the application from the database implementation. People with diverse roles and backgrounds interact with SQL and are well-versed in its peculiarities: from analysts to database administrators to web front end developers.

Despite this, for another example, there is no simple query that can 'insert this value for a key or update that key if already present' atomically. Some SQL implementations provide extensions for this elementary and very common task (like MySQL's ON DUPLICATE KEY UPDATE), and it is possible with stored procedures at risk of losing performance to exception handling.

SQL is designed for 'relational' databases: that is, each row in a table expresses a relation and so must logically be unique. The adherence to this concept is why SQL cannot answer the simple sales query, and why No-SQL databases are justifiable not just on grounds of performance and scalability: they often fit the problem domain better. When a design requirement fails to fit the use case it should be re-evaluated: relational databases are very handy for some sense of purity but as systems like Hive demonstrate, things more or less work without pure relational semantics.

SQL imposes weird design constraints on a general purpose database: people add dummy 'id' columns to give each records a relational uniqueness. As with all failed designs it's true that relational databases have real advantages in many cases, but the choice to demand these strict semantics should lie with the user, and a standardisation of the syntax for avoiding them would mean that SQL could deliver on its promise of portability across database implementations.

We all benefit from a common language, and tying SQL to one database implementation dogma inspires the proliferation of No-SQL mini-languages, each with a learning curve and lacking features. It's time to wrest the familiar syntax from the constraints of an ultimately failed design and admit that non-relational No-SQL techniques have real benefits to deliver.

Post a comment

Judging innovative software

Posted 2015-06-16 04:00:00 GMT

There's an old aphorism: execution matters more than ideas. In software I think that's very wrong — I'll elaborate but the question here is how can you evaluate an idea for a piece of software before it's implemented and in production testing? It's definitely possible to a certain extent, and this is an important skill.

Firstly, let me define what I mean by an idea. I want to differentiate between ideas and desired outcomes. An inexpensive autonomous flying car or a wonderful app that can transcribe your thoughts are both exercises in wishful thinking. They're science fiction, indubitably of immense value if they could be created, but definitely there is no clear path to an implementation. An idea in software is a method of implementation, something like trace compilation or the Bitcoin blockchain.

A software idea rarely enables some new capability. Generally there is a way to replicate the software's function in some other way, for example, by paying people to do it manually, or by constructing specialised physical machines. A software idea is about changing the balance of resources needed to achieve a capability. For example, with trace compilation, you can get the benefits of explicitly typed machine code without having to do costly static analysis. A software idea is generally about performance, albeit potentially about a huge shift in performance characteristics (e.g. enabling large-scale de-centralised trustworthy but anonymised financial transactions).

Is it worth investing the development effort in a new software idea? When you come up with a new idea, people will inevitably attack it. As Ben Horowitz says, Big companies have plenty of great ideas, but they do not innovate because they need a whole hierarchy of people to agree that a new idea is good in order to pursue it. If one smart person figures out something wrong with an idea–often to show off or to consolidate power–that’s usually enough to kill it. How can you, the inventor, yourself decide if your idea is worth investing your time in further developing, when you and others can find issues with your new scheme?

There are classes of attacks on any new idea, that are essentially more about newness rather than the idea. For example:

— it's not been done before [there's an inexhaustible supply of inertia, entropy and lethargy in the world]

— it will be hard to manage operationally [only if you for some reason deliberately choose to not develop the necessary production monitoring tools]

— it will not work in production at a specific scale - without any actual issue being identified [quite insidious, because to counter it, you'd have to develop the project sufficiently that it could be put into production]

As most new software ideas are experimented with or thought about in people's free time, and then to bring them out of the whiteboard stage a huge amount of effort is needed, these attacks can stifle a project immediately. I believe they should be disregarded as much as feasible and instead the discussion should center on the idea itself, rather than on the issue of its novelty.

A very valid reason to dismiss a project is the existence of an alternative method with better performance. Quantitative estimates are essential here. [One way to strangle a project at birth is to require such detailed projections that it must already exist before its creation can be justified.] Beyond this first order inspection, Hints for Computer System Design by Butler Lampson illustrates a series of practical considerations.

The cost of development of a system does (and should) factor very much into the decision about whether to pursue it. This is unfortunately entirely dependent on the particular people who will create it. One trick is to force very short timelines for prototypes (hackathons, etc.) but that severely constrains the scope and there is a huge natural tendency for the offspring of prototypes to be coerced into production - casting doubt on the original implementor and the idea itself. Some people can give realistic estimates of development time and others cannot; take the best guess at the distribution of development resources that will be required to achieve a specific level of benefit.

Once you've thought and fought through the above, the actual implementation might be relatively straightforward. Note that generally a new idea uses a particular resource much more heavily than it was used previously. For example, a new image processing scheme might rely on CUDA GPU computations or the SSSE3 PSHUFB instruction, where before only the scalar CPU instruction set was used. This will inevitably cause unexpected interactions when deployed at scale by changing the system's characteristics (in this case for example by drawing more electrical power). The ability to handle these issues is a reflection of the degree of technical stagnation the wider system already faces (e.g. aging compilers, fixed JVM versions, etc.) and generally the necessary fixes will benefit even the old system. That sometimes makes the arguments about them easier to overcome.

Programming the actual implementation is relatively trivial once the broader picture has been set. The quality of the implementation should be easy to measure given the discussions around the quantification of the benefit of the new approach, and once measured things naturally improve - lighting the path is harder than following it, and ideas themselves definitely have a social value beyond their first implementation.

Post a comment

Grep orientated programming

Posted 2015-03-23 03:37:21 GMT

One key indicator of a software projects amenability to change is its greppability. Projects that are not greppable take longer to modify and discourage casual contributions - and casual contributions are valuable not only in the open source world but also in enterprise where the consequences of being hard to casually modify are exhibited in the emergence of competing solutions or meetings about trivialities.

What is greppability? Grep is the name for a program for searching text. Wiktionary defines greppable as a a format suitable for searching. I don't think this really captures the issue in terms of software where the source code is almost always in a simple text format - greppability is determined by its structure.

Greppability is the ease with which one can navigate a body of source code just by searching simple text keywords. From determining what code caused an output to tracing all the callers of a function, there's plenty that can be possible by text searching - or not depending how names are used or the project structured. And the easier it is, the faster and more reliably new developers can be productive.

For example, a Microsoft style error message might be Action Failed Error Number: 2950 - this is incredibly ungreppable. In a large codebase: the words Action Failed or Error Number are likely to occur very frequently all over and even the number 2950 is likely to appear often. Therefore even a very skilled developer with full access to the source code, will on encountering this error will have a great deal of difficulty in determining the place where it was generated. A highly greppable alternative for this would be to include a distinctive keyword like access_macro_vba_fn_error in the message - this hopefully will appear only in places in the source code that are relevant.

Naming is obviously a key issue. Don't take advantage of separate namespaces to call different things by the same string - if a function is called getName then searching for it is likely to throw up hundreds of unrelated hits to a quick text search. But if it were called something more specific getWidgetName then instantly it's easier to figure out what is calling it and consequently the ramifications of changing its behaviour - reducing the incidence of unpredictable bugs.

Constant indirection is the enemy of greppability. It might be that the MS codebase (that I haven't seen) has something like const int kErrorAccessVBAMacroFun = 2950 in it. Once you've found 2950 is tied to this constant, then you have to grep again for the name of this constant to find out where it is used - making the process tiresomely more convoluted.

As a Lisper, it's sad to admit but dynamic code generation, introspection and macros can be the enemy of greppability. For example, there could be a DEFINE_ERROR(AccessVBAMacroFun, AccessErrorBase + 50) or something that would mean anybody grepping for kErrorAccessVBAMacroFun would have a hard time tying it to 2950. There might be an awesome error database tool but how will someone new know about it? While it might seem like good separation of concerns and neat code, it's not greppable. The fact that a new developer, however skilled, can't easily figure out which software caused the effects if he or she came at it from its external output is bad in itself.

Take a minute to think about greppablity - with a clever code generation or dynamic database scheme, try to have some keyword or string from the generated output appear in the source code, maybe just in comments.

Making code more greppable can have no cost, but opens up another tool to people on the project, and a tool specially easy for unsophisticated people or simple automation to use. Grep for the win!

This is actually why I think it's a good idea for error messages to contain a short identifier that can be searched for. This helps not just when searching the source code for that message, but also, for example, when searching the web for others encountering the same error, possibly with a differently worded message due to internationalization and localization. As a bonus, this gives you a convenient identifier for adding internationalization and localization to your software with something like gettext. This also extends to warning messages and other types of messages, but it's especially useful for error messages, as those are the most common kind of message for which one would want to find out what caused the message to appear and what can be done to prevent that condition from occurring.

Posted 2015-03-23 17:27:26 GMT by inglorion

my thoughts exactly!!!

Thanks for putting it out there...

Posted 2015-03-25 10:05:54 GMT by Anonymous from 148.87.67.201

Fremlin, instead of writing this on your blog, an appropriate response would entail three words and one exclamation point.

Posted 2015-04-10 10:07:59 GMT by Anonymous from 24.160.38.124

Post a comment

The Right Price for the Price is Right: Optimal Bidding

Posted 2015-01-25 03:13:38 GMT

The famous TV game show, The Price is Right had an excellent one-bid game where four players would take turns to give a guess for the price of an item (no guesses can be repeated). Any guess higher than the item price (an overbid) was discarded and then the closest remaining lower guess would determine the winner. Sometimes there was an exact prize bonus too.

This game has been studied extensively. When I first heard the rules, I wrongly intuited that the overbid condition would make it easier for the first player: quite the opposite.

The first simplification to make thinking about the problem easier is to reduce it to assume that all participants share the same model of the distribution of the price. Provided they are not sure about the price, then this resolves to guessing the closest value under a fair many sided die or roulette wheel spin.

Suppose we first consider the case where if all contestants overbid, then nobody wins the prize and the game is not repeated. The strategy of the last player is very easy: just pick the one more than a previous bid, or pick the lowest possible bid, whichever one has the highest probability weight between it and the next highest bid or infinity if it would be the highest bid.

Inducing backwards, the second from last player must consider that the last player will take almost all the probability mass from his bid, by bidding just above it, if that bid has the largest probability region above it. Therefore he or she must choose a bid that leaves another more tempting region for the last player to take. This implies that each preceding player must take at most their fair share - that is the first player must bid at min{b : P(X ≥ b) ≤ 1/n}, or for four players on a roulette wheel from 1-36, he or she would bet just exactly than 1/4 from the end, or 28 (winning on the 9 numbers, 28-36).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

Suppose he or she bid anything less - even just one less, 27. Then he or she would have 10 winning numbers, but the next player might be tempted to take the range 28-36 which has 9 winning numbers and leaves the big region 1-26 to be safely split between the third and fourth.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

Otherwise bidding below, the lowest the second player could safely go is 18, and also only get nine numbers. As there is no advantage to this, he or she might as well go for 28.

We therefore assume the first player picks 28, so the second player will take 19-27 (nine numbers), leaving the third player to take 10-18, nine numbers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

Consequently everybody will get an equal share. This analysis differs extremely from The Price Is Right, But Are the Bids? An Investigation of Rational Decision Theory by Berk, Hughson, and Vandervonde. In their version they fail to consider the discrete nature of the problem and assume that in cutting off another player (guessing just above another player) one receives the entire probability mass of that player. However, of course, one cannot repeat the exact guess so there is a little that previous player gets to keep.

As another difference, they consider a game where in the case where everybody overbids, the bidding is restarted (with an implicit maximum of the previous lowest bid, and in the same player order). In this version, the last player has an incentive to not bid 1, as the game will then restart. Then for all possible prices outcomes below the lowest bid, the last player will receive a probability mass Plastwin that he or she would win after a restart. This gives a probability mass bonus of PlastwinP(min(bi) > X) to the interval obtained by cutting off another player. Assuming as this paper does, that the space of bids is continuous and not discrete this means that the last player would cut off the best previous player's interval (so in this continuous bid version where each bid has measure zero, taking the entire probability mass) getting at least 1/3 of the mass if not overbid, and otherwise gambling on a restart. This implies that Plastwin is 1/3.

In practice, contestants are very nervous about overbidding and in 54% of cases the winning last bet is to cut off the highest bid.

Post a comment

ToqPiq: XKCD comics on the Qualcomm Toq smartwatch

Posted 2014-11-02 23:43:56 GMT

During the YC Hacks Hackathon, I made an app for the Qualcomm Toq SmartWatch that shows XKCD comics on the watch. It's pretty rough and ready.

Sad that the API from Qualcomm does not include more functionality (doesn't even allow arbitrary aspect ratios for the photos shown on the watch!). I've thrown the hack on GitHub, have fun!

Post a comment

How to scale a consumer web product launch (load shedding)

Posted 2014-11-01 21:46:09 GMT

The situation: you've come up with a cool new app, have beta-tested it for a while and are ready to open it up to the public. How much server capacity will you need? There's no way to estimate the loadspike - one hit post on reddit or a partner promotion could get you 100k uniques over a few hours and if you've got the killer app you think you have that might just be the start of the onslaught. To control the load, you can have an invite flow where you gather customer contact information and then notify them that they can use the product, but this is likely to cause more of a drop-off than immediately wowing people.

There are three engineering problems of scale here: the obvious one is making the software you run efficient - modern servers can handle 10k+ rps per core - for efficient requests. This requires monitoring and measurement. The second obvious one is ensuring a good asymptotic response to additional hardware resources - adding servers should make it possible to serve more customers, but if you have a single database shard that is authoritative then adding more API servers might not help at all. Again, monitoring and measurement are essential to verify that the characterization of the system fits the model you have of it under the real load it experiences.

The third problem is making sure that in the event of a failure of the scale plan, that the user experience is still professional. For example, if things are overloaded or broken, disable marketing campaigns and notifications drawing people to the product. This may seem obvious but its easy to forget about Adwords when product is down.

The minimal example of a load shedding UX is the famous Twitter fail whale - a static landing page (served perhaps from separate reliable infrastructure) that you can redirect people to. Shifting load can actually work well for highly motivated users: if someone really wants your product then they will come back tomorrow, so downtime mostly loses the chance to convert people who were just curious into customers. To try to minimize the loss, the landing page can ask for contact information so that you can notify people when the service is fixed. Having a count of customer sign-ups might help turn negative publicity around downtime into a story about popularity.

Depending on the product, it may be possible to implement a less debilitating load shedding: request fewer results from the server, do less ranking, depend on client-side caches and reduce polling frequency. Plan for this by changing client code to not retry aggressively and have server-side timeouts on queued requests: if a request is in a queue for longer than the customer is likely to wait for it, just fail it rather than spending compute resources.

Even with a good plan, and a well characterized system whose response under load is well understood, an influx of publicity will bring a new class of semi-malicious or malicious use. People will poke around at your API and use unexpected classes of devices to access the product. This is one example demonstrating that auto-scaling can be a really bad idea: if someone is DDoSing you, then adding scale to handle that just makes the DDoS more expensive. Some mischievous attackers specialize in finding poisonous requests that will consume many server resources while very few client ones. Be ready to react with reliable dashboards to give you the information you need, and do not depend on automatic systems. A load spike can turn into a delightful opportunity to exercise your disaster recovery plan when servers overheat and lock up :)

Post a comment

Redundant video bitstream

Posted 2014-10-22 17:47:57 GMT

Video has the most computationally heavy compression in widespread use. At the same time, layer after layer of abstraction with a view to incredible flexibility, where each group dreams of a container format that will carry any codec, leads to an incredible duplication of metadata.

For example, how many times is the video resolution encoded in an MP4 file with a typical MPEG4 AVC payload? The instinctive reaction of each group of architects at each point in the stack is to redefine the description of the video in some slightly inadequate schema that is not quite a superset of the others, so all are needed. For VOD or realtime video, the resolution is eagerly repeated again and again in each extra transport layer. Certainly for a standard MP4 stream, the resolution must be in the H.264 Sequence Parameter Set used by the video codec. But then also in the avcC box - the configuration of that codec in the MPEG4 stream format, and then also in the track header tkhd box. So I count at least three. Are there more?

Why is this bad? The information is unnecessary, in aggregate over the billions of videos made in the world it wastes huge amounts of not only video storage but mental bandwidth in standards texts, and a video file can now be created that is inconsistent so every implementation needs error handling for this case.

This is, I suppose, an example of a sort of bikeshedding: an issue so trivial (the resolution of a video is easily understood) so that each abstraction is eager to take responsibility for it, whereas real practical issues, like the relative placement of the indices inside the file, are left to fall through the cracks.

Post a comment

Activating the Dynamixel MX-28T actuator

Posted 2014-07-29 16:59:19 GMT

In our quest to make a robotic hand, we procured an awesome actuator, the Dynamixel MX-28T.


This marvelous device can not only spin but also go to a precise goal position, and hold it against varying torque, using its mighty 72MHz processor to adjust as necessary.

We wanted to use the usb2ax to control it. To our surprise this is actually impossible out of the box, as oddly enough, the MX 28T is programmed to start off listening at 57142.9 baud and the usb2ax can only use 1MHz. This is all the more bizarre because the base clock serial clock of the MX 28T is 1MHz or 2MHz depending how you think about it.

I guess we could have tried to do something cool with a serial port as this is quite close to 57600, but we didn't and we got another controller to program the poor MX-28T which lives on a nice bus to change register 4 to value 1 so we could talk to it.

Then everything was hunky dory. The servo is surprisingly strong!

Post a comment

The mmap pattern

Posted 2014-04-30 16:38:37 GMT

There are many choices in software engineering that are visible only to the developers on the project: for example, the separation of responsibilities into different parts of the program are (hopefully) invisible to the user. These choices can descend into a question of personal taste and I personally lean towards simplicity. My experience has shown that people tend to create complex generic interface hierarchies, only to have them hide a single implementation. What's the point in that? On the other hand, there are architectural decisions that affect the way the program behaves. For example, whether processing occurs on a server or mobile device ends up changing how the system can be used.

I want to bring up an underused architectural choice: the defined memory layout persisted to disk by the operating system via mmap. All malloc memory on modern UN*X systems is mmap'd. But explicitly choosing a file and mapping that into the memory space of the program means it can persist across crashes. And it doesn't just persist in terms of storage on a backing device, but the OS is aware the pages are mapped into memory, so on start-up there will be no disk read, no serialization delay and everything is ready. In many systems the effort of reading in data, whatever trivial transformation is being performed, can be extremely costly in terms of CPU time (instructions and dcache thrashing). With a mmap'd file, this time can be reduced to nothing.

One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption. The benefit here is that the system can be developed rapidly with code reviews focusing on data integrity and paying less attention to complex interactions that can lead to crashes.

Modern CPUs have the ability to arbitrarily (generally at a 4kB granularity) map memory addresses as visible to a program to physical RAM addresses. The technique I am advocating here is a way of exploiting this hardware functionality in conjunction with operating system support via the mmap call (that turns a file into a range of memory addresses). It is quite possible to share mmap regions across processes so this gives a very high bandwidth unsynchronized interprocess communication channel. Additionally, the regions can be marked read-only (another nice capability afforded by CPUs) so data-corruption failure cases can be avoided entirely.

The main implementation difficulty with using a mmap'd region is that pointers into it must be relative to its base address. Suppose one were to try to persist a linked list into such a region. Each next pointer in the list is relative to the base address of the region. There are multiple approaches: store the base address in a separate (maybe global) variable, and add it each time, or try to mmap each region to a well-known start address (and fail if it cannot obtain that address). Generally with some trickery it is possible to exploit the memory remapping capabilities of the underlying CPU to reduce this overhead (i.e. store the base offset in a segment or other register). Each of these alternatives has advantages and disadvantages which can be debated; in practice, once the idea of persisting state to mmap files is introduced into an architecture, there are various reasons to try to use multiple regions (e.g. to support fixed-sized and non-fixed-size records, and to enable atomic exchange of multiple versions of the state).

Though there are low-level opportunities, this technique can actually be extremely beneficial in garbage-collected scripting languages where dealing with large amounts of data is generally inefficient. By mmap'ing a region and then accessing into it, the overhead of creating multitudes of small interlinked items can be reduced hugely. Large amounts of data can be processed without garbage collection delays. Additionally, the high cost of text-processing can be paid just once, when first building up the data-structure and later manipulations of it can proceed very rapidly, and interactive exploration becomes very convenient. The instant availability of data can reinvigorate machine learning work-flows where iteration speed from experiment to experiment is a constraining factor.

Despite the advantages, this technique is not widely exploited, which is why I'm writing about it. For Lisp there is manardb, and in industry there are several very large systems of the order of petabytes of RAM which use this idea heavily. Consider it for your next project!

Damn, this was trivial using PL/1 on Multics back in the mid 70s.

Posted 2014-05-01 05:56:25 GMT by Anonymous from 174.17.212.195

MSVC has support for __based pointers, maybe there is something like this in gcc / clang?

http://msdn.microsoft.com/en-us/library/57a97k4e.aspx

Posted 2014-05-01 07:11:48 GMT by Dimiter "malkia" Stanev

With modern 64/48 bit memory space, could the application choose *exactly* where to map the file, thereby avoiding the need for base pointers or other fixups.

Posted 2014-05-01 09:13:15 GMT by Chris Dew

"pointer swizzling" is one term for the fixup technique. The Apple Newton's OS is the only major user I can think of, what are some others?

Posted 2014-05-01 10:08:23 GMT by abrasive

Good read. I'm a big fan of using mmap(). One thing to note is that you need to be careful with your file formats so they are aligned properly.

A good way to work around this is to preprocess files for the specific architecture it's currently on. Key points to keep in mind are native type width, alignment.

Posted 2014-05-01 11:22:24 GMT by Steven

'''

There are multiple approaches: store the base address in a separate (maybe global) variable, and add it each time, or try to mmap each region to a well-known start address (and fail if it cannot obtain that address).

'''

ASLR is a conspiracy created by "security" professionals to make programs run slowly. Eschew it~! : )

Posted 2014-05-01 14:15:03 GMT by babycakes

Yes, a standard implementation of mmap can put the memory at a fixed address. It's got a lot of options; see the man page. This doesn't even require more than 32 bits of address space, just some coordination with your linker to ensure that the region you want to use for mmap'd pages is not otherwise used. In fact, this is essentially what the runtime linker/loader does when you load a dynamically linked library.

Posted 2014-05-01 14:41:49 GMT by Anonymous from 174.52.89.43

"Despite the advantages, this technique is not widely exploited"

Are you sure? Image-based systems such as Smalltalk and some Lisps have been around, if not very common, since the 70s. Your suggestion seems to be very closely related.

"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption."

This might indeed be true for events such as abrupt power failure. However, if the crash is due to corrupted memory, you would not want to restart from such a damaged state. I don't see that there's an obvious way to differentiate these two cases.

Posted 2014-05-01 15:29:45 GMT by Michael Schürig

Hello John!

Could you explain manardb behaviour if i migrate from 32-bit to 64-bit SBCL or linux distribution? How about class redefinitions (add or remove slot)?

Posted 2015-05-27 00:40:48 GMT by Anonymous from 46.191.216.235

Post a comment

Costs of energy from different sources by kWh

Posted 2014-04-24 17:01:43 GMT

How to compare the cost of energy? Each different source is traditionally traded in different units (kWh, therm, gallon). Here I take the most recent information from the Bureau of Labor Statistics for San Francisco, Oakland, San Jose (the Bay Area) and digest it into kWh.

Energy source Cost per kWh in USD cents Relative cost Efficiency estimate Cost per usable kWh Notes
Natural gas (utility) 4.5 1 25% (18?) 29.3 kWh per therm, efficiency estimate based on CNG vehicle which is unfair comparison
Petrol (gasoline) 11.1 2.5 25-30% 44.4 33.4 kWh per gallon
Electricity (mains) 22.1 4.9 90% 24.6 Most convenient source of energy!

The cost of obtaining energy from solar installations is becoming competitive with grid electricity but is still much more expensive than other energy sources (particularly natural gas!).

Deciding in practice between which energy source to use is tricky. For example, if you have a plug-in hybrid car, should you charge it from the mains or from a petrol station? That depends on its efficiency per mile traveled from the different sources (normally much higher for electricity) and also on the charging efficiency, as optimistically 10-20% but sometimes even more of the mains charge is wasted and not stored in the battery.

EDIT 20140424: Added estimates of usable energy in terms of efficiency of a motor. For heating, much higher efficiency can be obtained from the cheaper sources like gas (e.g. some people estimate 70-80%).

Post a comment

Older entries (91 remaining)