The crash did not lose a paragraph. It gave the agent a personality disorder.

As I argued in a previous post, a crash mid-turn is the normal operating condition for an agent that acts, not an edge case. Vendors have answered by selling “durable” and “stateful” agent runtimes. Most of them made the agent retryable. Almost none made it continuable. The waffle-maker amnesia is the symptom that tells the two apart.

• Retryable: you can re-run the whole turn from the beginning and get a correct result.
• Continuable: you can resume the turn from where it died.

The relationship is exact: continuable = retryable + durable intermediate state. The only added ingredient is durability of the in-flight computation. Everything below is about the price of that one ingredient.

Where you paid for durability

The usual way to frame this trade is latency versus correctness. That is the wrong axis. The right one is where in the computation you paid for durability: at the ends, or in the middle.

Retryable needs durability only at the ends. Two things must survive a crash: the input, and the effect. A durable input and an idempotent sink. Everything between them can be as unreliable as you like. The container can lose all its RAM and vanish, because a retryable turn is all-or-nothing: nothing in the middle was ever externalised, so throwing the half-finished run away and restarting from the durable input is not a compromise, it is correct. This is why you can bolt retryability onto any disposable executor: put a queue in front, make the consumer idempotent, treat workers as cattle. The cost is paid in latency and pacing. You cannot stream a result you might have to retract, so you buffer everything until the turn commits and release it at once.

Continuable needs durability in the middle, a journal of in-flight progress written before the crash, not after. You cannot throw the run away, because you have already externalised part of it: a message the user has seen, the first half of a recommendation. To resume instead of restart, you must have written down what you did as you did it. There is no queue you can put in front to get this for free. You change what the executor records while it runs.

Why the cheap trick does not work on an LLM

Durable execution is not new. Temporal, Restate, DBOS, and Cloudflare Workflows all make the middle of a computation durable, and they do it with the same trick: journaled replay. The workflow records the result of each step as it completes. After a crash it re-runs the workflow code from the top, but every step that already has a journal entry returns its recorded result instead of executing again. Replay re-derives the program’s state cheaply, because the code between steps is deterministic: given the same journal, it walks the same path.

An LLM is not deterministic. You cannot replay the prompt and trust it to re-derive the same generation, and that non-determinism is the whole reason you reached for a model in the first place. So the cheap half of the trick, replaying the recorded steps and re-executing the deterministic glue between them, breaks the moment the glue is a model call.

The fix is one the durable-execution world already knows: model each LLM call as an activity. An activity is journaled once when it completes and never re-executed on replay; its recorded output is authoritative. For a deterministic workflow the journal is cheap, because most of the state can be re-derived and only a handful of external results need storing. For an LLM you have no choice but to store the actual generations, every token the model produced, because there is no code path that re-derives them. The black-box, non-replayable executor is precisely what makes continuability cost real durability instead of cheap replay.

And that journal, the record of every decision the model committed to during the turn, is exactly the memory the amnesiac agent was missing. The waffle-maker bug is not a separate failure. It is what ends-only durability looks like from the user’s chair: the model’s accumulated decisions lived in container RAM, the container went away, and there was nothing on disk to continue from, only the input to redo.

It is not the delivery guarantee

One objection worth heading off. Both models deliver effects the same way: at-least-once delivery into an idempotent sink, which nets out to effectively-once. The delivery guarantee is not what separates them.

The difference is when you are allowed to externalise an effect. Retryable buffers every effect to the end of the turn and releases them together, so a crash before the commit point erases a run that never touched the outside world. Continuable externalises incrementally as the turn proceeds, which is what makes streaming and mid-turn actions possible, and pays for that by journaling each decision before it acts, so a crash leaves a record to resume from instead of a void to re-guess.

The question to ask any substrate

You do not shop for continuability as a checkbox on a framework’s feature page. It is a property of the storage sitting under the loop, so the only useful thing to do when you evaluate an agentic LLM substrate is ask one question: when the model commits a decision in the middle of a turn, does that decision land on storage that outlives the container before the next crash? Put it more bluntly. Is my disk continuable?

That question is unkind, because a continuable disk is one of the least pleasant things an ops team has to operate. The journal sits on the hot path of every turn: each decision is written before it acts, so the write latency is added to every step the agent takes. To be worth trusting it has to be genuinely durable, with POSIX fsync semantics, where a write that reports success is actually on the medium and not in a buffer the crash eats. You want it colocated with the compute, ideally a local mount, so a journal write is not a network round-trip bolted onto every decision. And it still has to survive the node it is colocated with going away.

Fast, POSIX-durable, colocated, and crash-surviving, all at once: that is nightmare territory. Network-attached block storage gives you durability and a detach latency you feel on every write. Local NVMe gives you speed and dies with the instance. A distributed POSIX filesystem gives you all of it and a pager that never sleeps. Persistent disks are the bane of every ops team for exactly this reason, and continuability puts one on the critical path of your agent loop.

So the real recommendation is not “make your agents continuable.” It is: decide whether you are willing to run that storage. If you are not, you are choosing retryable, and that is a legitimate choice with its own bill: buffer every effect to the end of the turn, give up streaming and pacing, re-run from the input on every crash. Neither option is free. Retryable pays in latency and pacing; continuable pays in storage operations. You can have retryability on unreliable execution. You cannot have continuability without durable execution.

Two durable surfaces instead of three

When I first started building agents, I used a substrate that runs on unreliable disks. It throws RAM away on a whim and promises me nothing like a fast, colocated, crash-durable journal, which is the one thing continuability cannot do without. So I did not get to choose continuability. I built the retryable system on purpose, and spent the design effort on making it need as few durable surfaces as possible: two, not three.

Here is the shape. During a turn the agent is not allowed to send anything. It can only propose messages: accumulate the set of outbound messages it intends to send. When the turn completes, that whole set is sealed into a Durable Sink in a single transaction, and the same transaction advances the inbound message’s lifecycle out of processing. From then on the Durable Sink owns delivery: it sends each message, and it marks the inbound message replied only once every outbound message is confirmed out.

Count the durable surfaces. The inbound message that arrived is one, the durable input. The Durable Sink that holds the sealed set and drives delivery is the other. The middle, every decision the model made during the turn, is never written down. It lives on local disk and dies with the container, and that is fine: a crash before the seal strands the inbound message in processing, where a sweep picks it up and re-runs the turn from the input. Nothing was externalised, so re-running is correct. That is the retryable bargain paid in full: two durable surfaces, an unreliable middle, and a re-run whenever the middle is lost.

A continuable version would add the third durable surface, the journal in the middle, and with it the fast, colocated, crash-durable disk I do not have. I traded that away for the retryable tradeoffs: no streaming, buffer to the seal, re-run on crash. Let us see how it does in the real world.

For a chat assistant, a crash mid-turn is an annoyance. You lose a paragraph and the user retypes their question. For an agent that takes actions in the world, a crash mid-turn is the normal operating condition, not an edge case. The moment an agent charges a card, sends money, or books a seat, “just run it again” stops being free.

The mechanism that prevents the second ticket has a name: idempotency. An idempotent operation can be applied many times and produce the same effect as applying it once. The uncomfortable part is that, across the integrations agents actually touch, most of this does not exist yet.

Why retries are not optional

Agent runtimes retry by design. A turn is a long chain of model calls and tool calls, and any link can fail: the model times out, a tool returns a 503, the container is rescheduled, the orchestrator redelivers a queue message it is not sure was processed. Every one of these ends in the same place — the runtime runs the turn again to make progress.

That is fine when the tool call is a read. Fetch the weather twice and you have wasted a few milliseconds. It is not fine when the tool call has a side effect on a third party you do not control. The runtime cannot see inside Stripe or Gmail. It only knows that it sent a request and did not get an answer. It has no way, on its own, to tell “the charge did not happen” apart from “the charge happened and the acknowledgement was lost.”

This is the classic exactly-once problem, and distributed systems people will recognise it immediately. What is new is where it has surfaced: not between two services you own, but between an autonomous agent and an open-ended set of APIs you did not write.

The matrix nobody wants to fill in

The shape of the problem is a grid. Down one axis, the integrations: Stripe, Gmail, Amadeus, Twilio, your CRM, every API an agent can reach. Across the other, the operations: charge, refund, send, book, cancel, post. Each cell — every (integration, operation) pair — is a separate exactly-once problem, and each has to be solved on its own terms.

For any given cell, three questions decide how hard it is:

• Does the partner give you an idempotency key? If you can attach a client-generated key to the request and the partner promises to deduplicate on it, the cell is close to solved.
• Can you ask whether the action already happened? If not, you can query after a failure and reconcile instead of guessing.
• Does it need a human in the loop anyway? Some actions are high-stakes enough that automatic retry is never acceptable.

A few illustrative cells make the spread obvious.

Stripe, charge. This is the good case. Stripe lets you pass an Idempotency-Key header, holds the result of the first request against that key, and replays it on retry. Send the same charge twice with the same key and you get one charge and two identical responses. Solved — because someone at Stripe did the work.

Gmail, send. Harder. There is no first-class idempotency key for “send this message.” You can lean on the Message-Id header and dedupe on the receiving side, or check Sent before retrying, but now you are reconciling against state that may not have settled yet. The cell is reachable, but you are building the guarantee yourself.

Amadeus, book. Harder still, and the one from the opening. Booking touches inventory and money, the confirmation may arrive seconds after the reservation is made, and a naive retry double-books. Here you are stitching together query-then-act with whatever idempotency the API offers, and getting the failure modes wrong costs a real ticket.

The grid is not small, and it grows every time a new API ships. Most cells are unsolved.

Why this stays unsolved

The reason is not that any single cell is intellectually hard. It is that there are thousands of them, and each one needs expert, per-API failure-mode analysis to get right. What does this partner do on a duplicate? Does the idempotency key expire? Is the “already exists” error safe to swallow, or does it sometimes mean something else? This is careful, unglamorous work, and it does not generalise from one integration to the next.

It is also work you cannot hand to the model. You cannot have an agent decide, at runtime, whether sending $500 twice is safe. The whole point of the exactly-once layer is to be the deterministic floor the probabilistic system stands on. A guarantee that holds “usually” is not a guarantee.

So in practice teams pick one of three options, none of them satisfying:

1. Accept duplicates. The default, because it requires building nothing. Fine for idempotent-by-nature actions, quietly dangerous for anything financial.
2. Require human confirmation. Safe, and it kills the autonomy that made the agent worth building.
3. Build the exactly-once infrastructure. Correct, expensive, and a one-time cost paid per cell.

The market is telling on itself

There is a tell worth noticing. Integration platforms have started marketing idempotency as a feature — Composio, for instance, talks about it — well ahead of shipping it as a real, per-cell guarantee in the API. When the marketing runs ahead of the product, it usually means demand has arrived before the capability has. That is exactly where this is: everyone building agents that act has hit this wall, and the layer that would fix it has not been built.

Someone is going to build it properly — a per-integration, per-operation exactly-once layer that an agent runtime can lean on the way it leans on Stripe’s idempotency key today. Until then, the answer to “what happens if the agent crashes mid-booking” is, for most cells, two tickets.

What I can help with

I’ve kept a real engineering job while moving between time zones and bad WiFi, so I know where this breaks down. Mentoring sessions usually cover one of a few things:

• Staying productive while moving. Time zones, unreliable connectivity, and the discipline of shipping when your day job is in another country. The hard part isn’t the travel; it’s keeping output steady when nothing about your setup is.
• The engineering work itself. Algorithm and system design, working through a problem you’re stuck on, or a code review of something real you’re building.
• Career questions. What to learn next, how to read the job market, whether the role you’re in is the one you should stay in.

I don’t do generic pep talks. Bring a concrete problem and we’ll work on it.

Why take advice from me

I do this for real, not in theory. Over three years and ~60,000 miles we drove the entire Pan-American Highway, from Ushuaia at the bottom of Argentina to Tuktoyaktuk on the Arctic coast of Canada — the whole length of two continents. The whole time I was the maintainer of SecureDNA. So when I say you can hold a serious engineering job from a truck with bad WiFi, I’ve done exactly that. You can follow the travel side at ElTruckito.com.

Booking a session

If any of this is useful to you, book a session with me on ADPList.org. Tell me what you’re working on beforehand so we don’t spend the call figuring out what to talk about.

The Initial Problem: Hosting on a Basement Server

When I first started my blog on WordPress, I had a rather unusual setup - the site was hosted on a server in a friend’s basement. While it may sound like a bit of a “ghetto” arrangement, it seemed like a cost-effective solution at the time. However, it soon became apparent that this setup came with its fair share of problems.

One of the biggest issues was the server’s reliability. For reasons that were never quite clear, the server seemed to go down regularly, without any good explanation. This meant that my site was often inaccessible for hours or even days at a time, frustrating both myself and my readers.

To make matters worse, the site’s loading times were notoriously slow. This was likely due in part to the server’s less-than-optimal location and setup, but it also meant that visitors to my site had to endure lengthy wait times just to access my content. And as we all know, in the world of online content, every second counts.

As my blog began to grow and attract more readers, the maintenance of the site became an increasingly daunting task. With the server regularly going down and loading times being so slow, I found myself spending more and more time just trying to keep the site running smoothly. This detracted from the time and energy I wanted to devote to creating new content and engaging with my readers.

Eventually, in 2020, I began to investigate the possibility of moving my site to a proper hosting solution, such as AWS. However, as I quickly discovered, this option came with a hefty price tag. The cost of a properly hosted server, combined with the need for a distributed database, made this option simply unfeasible for me at the time.

It’s not just about hosting, maintaining WordPress can be a headache. While it’s an amazing system, opting for a static website can offer several advantages. It’s faster, easier to maintain, and more secure, making it a smart choice for those who value efficiency and security.

In the end, I realized that something had to be done to address the ongoing issues with my site’s performance and reliability. After careful consideration and research, I made the decision to migrate my site to Jekyll, a static site generator that promised faster load times and a more streamlined maintenance process. And while the process of migrating my site was not without its challenges, I can now say that I am thrilled with the improved performance and ease of use that Jekyll has brought to my blog.

I chose GitHub Pages as my hosting solution, as I regularly use Git and GitHub in my daily work, and the added bonus of open-sourcing the content was a nice benefit for the community.

The Long and Winding Road of Migrating to Markdown - According to the Internet

When I stumbled upon Swizec’s article on migrating from WordPress to Markdown, I was initially excited. His straightforward breakdown of the process made it seem like a simple task that could be completed in an afternoon. “Pfft, an afternoon of work at worst,” he wrote, listing off the steps like they were nothing: export from WordPress, find a script, sip margaritas.

It all sounded so easy and straightforward, and I couldn’t help but agree with Swizec’s assessment. “That sounds about right,” I thought to myself, feeling confident that I could tackle this migration with ease.

But then, Swizec hit me with a dose of reality. “Suddenly it’s 6 months later and you’re losing your mind,” he wrote. His words were a stark reminder that even the simplest tasks can often become complex and time-consuming.

As an optimist, however, I refused to be discouraged. Sure, the process might not be as straightforward as I initially thought, but surely it couldn’t be that difficult, right?

So, being the lazy optimist that I am, I decided to explore my options. I turned to Fiverr, hoping to find someone who could handle the migration for me. I was shocked to discover that most of the quotes I received ranged from $1500 to $8000. That seemed like an exorbitant amount of money for something that sounded like it should only take a few hours.

One freelancer even warned me that it could take 2-3 months to complete the migration, although he optimistically suggested that it might only take a month.

The whole experience left me feeling a bit frustrated and overwhelmed. But despite the initial setbacks, I remained determined to find a solution that would allow me to migrate my site without breaking the bank or losing my mind in the process. And while it might take a bit more time and effort than I initially anticipated, I’m confident that I’ll be able to find a way to make it work.

What Made This Blog Different

Perhaps I was a bit fortunate that I had recently teamed up with Juraj to create a brand new website for Nomadic Workplace, my umbrella company that enables me to pursue my digital nomad lifestyle while remaining legally employed in the US.

When it comes to martinkysel.com, I can confidently say that it boasts two significant advantages over other blogs out there. Firstly, the vast majority of its content follows a similar pattern and covers similar topics. This consistency makes it easier to script a migration without having to deal with a lot of edge cases.

Secondly, all of the comments on the blog are saved in Disqus. This means that they can be easily transferred and integrated into a new platform, without the risk of losing valuable feedback and engagement from readers.

Finally, let’s be honest - the old blog design was pretty unappealing. So, any improvement in this area would be a massive gain for both myself and my readers. With the new Jekyll-based site, I was able to create a much cleaner, more streamlined design that not only looks better but also makes it easier for readers to navigate and engage with the content.

The Steps of the Actual Migration Process

So, how did the whole migration process actually go? Well, to start with, I discovered a migration tool developed by Lonekorean over on Github, which was almost perfect for my needs. However, there were a couple of essential components that it was missing.

Firstly, it didn’t have the ability to deal with [code] blocks, which seemed to be a non-standard feature of my old Wordpress site. This was a bit of a headache, but as is often the case with internet problems, I was able to find someone who had already struggled with and solved the issue in the past. In this case, it was a user called mxro, who had written some incredibly simple Typescript code to handle the preprocessing.

The second issue I encountered was with images. Thankfully, there were only a few scattered throughout the entire blog, so I was able to handle them all manually. While this was a bit time-consuming, it ultimately wasn’t a huge deal in the grand scheme of things.

Of course, the migration process wasn’t just about transferring content from one platform to another. I also had to select a new theme for the site, add some new icons, turn the banner into an avatar, and reset DNS entries. While these tasks weren’t necessarily difficult, they did require some careful attention to detail in order to ensure that everything was working smoothly and looking great.

A Surprisingly Short Migration Process

The entire process took only four hours - no joke. I started around 8PM and got so absorbed in it that I barely noticed the time passing. I remember making a fresh batch of argentinean mate while working away. By around 11PM, I decided the website was good enough and made the move to switch the DNS entries over to the new website. Of course, there may be some kinks to iron out over the next few days as I discover any issues that may have gone wrong or missing, but overall, it was nowhere near the month or six months I was anticipating. And if you don’t believe me, you can check the git history. The blog’s source is open on GitHub Pages.

Lessons Learned: Be Skeptical

Reflecting on this experience, I have come to realize that there are a few key takeaways that I believe are worth sharing with others.

Firstly, when it comes to using standard tools, the integration with existing systems can be seamless. In my case, because I was using widely accepted tools for the migration process, such as WordPress and Markdown, the transition was relatively smooth. However, the process could have been complicated if I had relied on more obscure or specialized tools.

Secondly, keep it simple stupid. Pick a system that requires the least maintenance.

Thirdly, it is important to approach time estimates with a healthy dose of skepticism. I was initially quoted a range of 1-3 months for the migration process by one service provider, which turned out to be completely off-base. It is essential to use common sense and T-shirt sizing estimates.

Lastly, be wary if your customer is another software engineer. Trying to “bullshit” your way through a negotiation will only lead to frustration and disappointment for both parties. Other engineers are highly skilled professionals who understand the complexities of building and maintaining software systems. They know what is feasible and what is not, and inaccurate estimates should alarm them to sketchy behavior.

The Benefits of Jekyll

There are several benefits to this migration: 1) Faster load times: Jekyll generates static pages, which means that there is no need to query a database or run server-side scripts to serve up content. This has resulted in much faster page load times and a better user experience for you, the readers. 2) Improved security: As Jekyll doesn’t require a database or server-side scripting, there is less potential for security vulnerabilities, which makes my life much easier. 3) Simpler design: Jekyll is based on templates, which has hopefully resulted in a more consistent and visually appealing design. This should make it easier for readers to navigate the site and find the information they are looking for. 4) Easier updates: Jekyll makes it easier to update the content on a website since it separates content from presentation.

Overall, the migration to Jekyll has resulted in a faster, more secure, and more user-friendly website, which should improve the overall experience for readers of martinkysel.com.

No More Ads!

I hope that the decision to remove ads from this blog will be a positive move for everyone involved. For readers, the absence of ads can result in a more enjoyable reading experience, free from distractions and slow loading times caused by ad content. Maybe this improved experience can encourage readers to spend more time on the site, engaging with content and potentially sharing it with others.

If you still wanna buy me a coffee, you can click the sponsorship button on GitHub!

The ads sucked anyway…

Simplified Looks

Simpler pages are typically easier to navigate because they present information in a clear and concise manner, with fewer distractions or unnecessary elements. When a page is cluttered or overly complex, users may struggle to find the information they are looking for, leading to frustration and potentially causing them to leave the site. The WordPress site was an old mess of fully of crappy elements from 2015.

By contrast, simpler pages tend to have a clear visual hierarchy that makes it easier for users to scan and locate the information they need. This was achieved through the use of white space, consistent typography, and clear navigation menus that now guide users to the most important content.

And man, this page really needed to be simplified!

Ability to File Issues

If you see a bug in any of the pages, just head over to Github and file an issue. I am happy to incorporate any and all feedback.

Breakage

If anything broke, please let me know! Either in the Disqus system or on GitHub.

Where to Learn More?

You can read more about the migration in my post From Ugly to Beautiful: The Transformation of Martinkysel.com.

WHY DO YOU CARE?

If you make a change to application code and something doesn’t work, you don’t want to have to deal with a complicated recovery. You want the database to be able to roll it back automatically so you get back to a working state very rapidly. Today, very often people don’t write code for databases, they have frameworks that do it (Hibernate, for example). This makes it impossible for a software engineer to write the code properly, or roll it back, because they don’t work on that level. When you make changes using rolling application upgrades, it is simpler, less likely to fail, and more obvious what to do when you do experience a failure.

With transactional DDL, it’s far less likely that you will have to deal with a partially upgraded system that has essentially ground your application to a stop. Partial upgrades like that may require your database administrator (DBA) to go in and fix everything by hand, losing hours of their time and slowing your software delivery down. With transactional DDL, you can roll back to the last working upgrade and resolve the issues rapidly, without taking your software delivery system or your application offline.

A SHORT EXPLANATION OF DML AND DDL

Essentially, DML statements are structured query language (SQL) statements that we use to manipulate data — as you might have guessed. Specifically, the DML class includes the INSERT, UPDATE, and DELETE SQL statements. Sometimes, we refer to these three statements as WRITE DML, and we call the SELECT statement READ DML. The standard does not differentiate between read and write, but for this article, it is an important distinction.

On the other hand, DDL is a family of SQL language elements used to define the database structure, particularly database schemas. The CREATE, ALTER, and DROP commands are common examples of DDL SQL statements, but DDL language elements may include operations with databases, tables, columns, indexes, views, stored procedures, and constraints.

Next, let’s start by defining a transaction as a sequence of commands collected together into a single logical unit, and a transaction is then executed as a single step. With this definition, if the execution of a transaction is interrupted, the transaction isn’t executed. Because a transaction must be ACID — Atomic, Consistent, Isolated, and Durable, that means that when a transaction executes several statements, some of which are DDL, it treats them as a single operation that can either be rolled back or committed. This means that you will never leave the database in a temporary, non-consistent state. Historically, databases haven’t provided the functionality of transactional DDL statements, but even today not all databases provide the functionality of truly transactional DDL. In most cases, this functionality comes with limitations.

Now, what does true “transactional DDL” mean? It means that all statements should be ACID, regardless of whether they are DML or DDL statements. In practice, with most databases, DDL statements break the transactionality of the enclosing transaction and cause anomalies.

A BRIEF HISTORY OF DDL

Originally, the idea of a data definition language was introduced as part of the Codasyl database model. CODASYL is the Conference/Committee on Data Systems Languages, and was formed as a consortium in 1959 to guide development of a standard programming language, which resulted in COBOL, as well as a number of technical standards. CODASYL also worked to standardize database interfaces, all part of a goal from its members to promote more effective data systems analysis, design, and implementation.

In 1969 CODASYL’s Data Base Task Group (DBTG) published its first language specifications for their data model: a data definition language for defining the database schema, another DDL to define application views of the database, and (you guessed it) a data manipulation language that defined verbs to request and update data in the database. Later DDL was used to refer to a subset of SQL to declare tables, columns, data types, and constraints, and SQL-92 introduced a schema manipulation language and schema information tables to query schemas. In SQL:2003 these information tables were specified as SQL/Schemata.

Transactionality of DDL, however, is not part of the ANSI SQL standard. Section 17.1 of ANSI SQL 2016 () only specifies the grammar and the supported isolation levels. It does not specify how a transaction should behave or what ‘transactional’ means.

WHY ISN’T TRANSACTIONAL DDL UNIVERSALLY PROVIDED?

There’s no reason why DDL statements shouldn’t be transactional, but in the past, databases haven’t provided this functionality. In part that’s because transactional DDL implies that DDL must happen in isolation from other transactions that happen concurrently. That means that the metadata of the modified table must be versioned. To correctly process metadata changes, we need to be able to roll back DDL changes that were aborted due to a transaction rollback. That’s not easy — in fact it’s a complex algorithmic task that requires the database to support metadata delta (diff), which corresponds to DDL changes within the current transaction of each connection to the database. This delta exists before the transaction is closed, and so it could be rolled back as a single transaction, or in parts in RDBMSs that support multi-level transactions or savepoints. Essentially, it’s not universally provided because it’s hard to do correctly.

For organizations moving to microservices, DevOps, and CI/CD, they have an essential new requirement — a database that supports online transactional DDL.

Let’s return to our concept of WRITE DML (update, delete, insert) vs. READ DML. You might ask yourself how do these statements collide in a system that supports multiple concurrent transactions and a DDL transaction is ongoing? Ideally the set of transactions that collide is as small as possible. A SELECT statement does not collide with an INSERT statement. There is no reason why this should be any different in the context of DDL. A DDL statement ALTER TABLE should not prevent a SELECT statement from executing concurrently. This is a common pattern in database design.

For DDL to be transactional you need to support multiple versions concurrently. Similar to multiversion concurrency control (MVCC), readers don’t block writers and writers don’t block readers. Without MVCC it’s hard to have transactional DDL. Traditionally, databases started with a locking system instead of MVCC. That implementation wasn’t suited to transactional DDL, which is why around 2005 there was a big shift towards MVCC — to provide concurrent access to the database, and to implement transactional memory in programming languages.

MVCC provides the semantics we might naturally desire. Read DML can proceed while conflicting writes (write DML and DDL) are executed concurrently.

WRITE DML AND DDL IN A LIVE SYSTEM

We have established that Read DML (SELECT) can happily proceed regardless of what else is executing concurrently in the system. Write DML (INSERT, UPDATE, DELETE) is not allowed to execute on a table that is being concurrently modified by DDL. Explaining the semantics of the conflicts expected behavior based on the Isolation Levels of all concurrent transactions is beyond the scope of this article.

To simplify the discussion, we state that both write DML and DDL are mutually exclusive if executed on the same resource. This results in operations blocking each other. If you have a long-running DDL transaction, such as a rolling upgrade of your application, write DML will be prevented for a long period of time.

So even though the DDL is transactional, it can still lead to database downtime and maintenance windows. Or does it?

ALWAYS ONLINE, ALWAYS AVAILABLE

The database industry is moving towards an always online, always available model. Earlier databases resulted in a message saying that something wasn’t available — that was essentially because you grabbed a lock in a database for a long period of time. That isn’t an option in an always online, always available model.

Customers, and therefore organizations, require applications to be online and available all the time. That means that transactional DDL is mandatory for the new world, and not only for the applications to run the way customers require them to. It’s also mandatory for organizations adopting DevOps and continuous integration and continuous delivery models (CI/CD). Specifically, that’s because without transactional DDL, applications developers cannot safely and easily make database schema changes (along with their app changes) online. That means that for organizations moving to microservices, DevOps, and CI/CD, they have an essential new requirement — a database that supports online transactional DDL.

I personally consider the term ONLINE misleading. The database is not offline while it holds a lock on a resource. A more appropriate term would have been LOCK FREE DDL. That is, metadata modification can happen without locking out concurrent DML.

THE AVAILABILITY VS. SIMPLICITY TRADEOFF

We said that write DML cannot happen concurrently with DDL to avoid ACID violations. Now, what happens to a system that has to be always up and needs to execute long-running DDL statements? Luckily enough, most DDL statements do not take a long time. Adding or removing columns from a table takes a constant amount of time, regardless of how long the table is. If the set of changes is small enough, it is OK to lock DML out for a short period of time.

But there are some DDL statements that need to process every row in the table and hence it can take a long time to process large tables. CREATE INDEX is a prime example of a long-running statement. Given that index creation can take multiple hours, it is not an acceptable option for an always online, always available application.

Specifically, for index creation, NuoDB and other databases implement an ONLINE or CONCURRENT version (I would have preferred to call it a LOCK FREE version). This version allows DBAs to create indexes without locking the table or requiring a maintenance window — an extremely important capability in a 24×7 availability world. However, these capabilities do not come for free. Online versions of common DDL statements tend to be slightly slower than their LOCKING versions, have complicated failure modes, and hard to understand ACID semantics. They also cannot be part of a larger multistatement DDL transaction.

Interestingly enough, sometimes the speed of execution is not the primary concern. In which case you may not want to use the online version. You might have an application that requires complex changes to a database schema and some LOCKING is a viable tradeoff for a much simpler upgrade procedure.

Atomicity, one of the four guarantees of ACID, states that: “Each transaction is treated as a single ‘unit,’ which either succeeds completely or fails completely.”

This becomes a very desirable quality if we think of transactions as a set of many DDL statements. An example would be: alter a table; create a log table with a similar name; insert a few rows to other tables. We already know that if DDL is not treated transactionally, you could end up with new rows in the other tables, but neither the CREATE nor the ALTER succeeded. Or you could end up with just the CREATE and no ALTER.

So, if you have an application that assumes that if there is a log table (the CREATE) it can also assume that the ALTER has happened, you might run into subtle bugs in production if the upgrade did not fully complete. Transactional DDL gives database administrators the ability to perform multiple modifications (such as the example above) in a single operation.

For developers, the strong Isolation guarantees of transactional DDL makes the development of applications easier. An application can only observe the database in state A (before the upgrade) or in-state B (after the upgrade) and will never see partial results. This reduces the required test matrix and increases confidence in the rolling upgrade procedure. Now that is easy to code against.

The tradeoff between simplicity of rolling upgrades that is LOCKING and the always available, always online that is NOT TRANSACTIONAL has been known to the industry since InterBase introduced MVCC to the commercial market.

CHOOSE TRANSACTIONAL DDL

Databases have changed a lot since 1959, and there have been many changes in customer expectations for user experience and application availability since then. Transactional DDL helps you avoid a scenario in which your application is no longer available, and gives your DBAs some peace of mind, knowing they won’t have to painstakingly repair the database to bring your software delivery back up to speed. Today, many databases offer transactional DDL, which will help you resolve immediate issues quickly by rolling back to the last working upgrade. In order to meet the always available requirements of today, choose a database that offers transactional DDL. But keep in mind that modern always-available, always-online applications require a database that not only simplifies upgrade scenarios, but also limits downtime due to long-running metadata modifications.

This article was originally published in The New Stack.

TL;DR: NuoDB is a distributed SQL database.

NuoDB is a SQL database, it is fully transactional, fully ACID, and fully consistent. This is a stark contrast to NoSQL solutions that have emerged in the last decade.

NuoDB is distributed, meaning that it is able to scale horizontally and automatically keeps multiple replicas in sync. Having multiple replicas of a database is a strict requirement for any service having 99.99% or better availability guarantees.

DO YOU RUN IN THE CLOUD?

NuoDB runs on all public clouds (AWS, Azure, GCP, etc.), on private clouds, and on-prem. We run in Kubernetes, Docker, VMs, bare metal, you name it. NuoDB allows you full flexibility to install the product anywhere you want while avoiding vendor lock-in with any of the cloud providers. I call our ability to run anywhere cloud-agnostic. Which is different from the next buzz word… multi-cloud.

DO YOU RUN IN MULTIPLE CLOUDS?

Being cloud-agnostic, we run in any of the clouds, but we also run in multi-cloud. Given that multi-cloud is a loaded buzz word, let me explain what multi-cloud means to us. We give our customers the ability to run one logical NuoDB database across multiple clouds, such as AWS and Azure, at the same time. Read about the KubeCon partnership announcement with Rancher.

The ability to run in multiple clouds pushes the boundaries of zero-downtime even further. Having a single logical database that spans two or more clouds allows your service to continue running even in the case of serious cloud provider failures.

WHAT ARE YOUR CONSISTENCY GUARANTEES?

Using the computer science term CAP, NuoDB is a CP database, meaning that we prioritize consistency over availability. Once NuoDB acknowledges a transaction, it is guaranteed to be stored in the cluster following all ACID principles. NuoDB is well suited for high-value data.

ARE YOU BUILT ON TOP OF ANOTHER DATABASE?

NuoDB is not build on top of another (open source) database such as MySQL or PostgreSQL. We have our own client, SQL engine, query planner, and query optimizer. As such, we have a slightly different SQL dialect from other SQL databases. We are compliant with the ANSI SQL standard. NuoDB has been around since 2010 and has had a chance to build our own custom SQL layer that is tuned to modern distributed environments.

DO YOU USE CONSENSUS/RAFT?

The short answer is no. The longer answer is: yes, but only for configuration management as part of the administration tier. Consensus for data changes does not scale well, which is why databases that are based on it combine it with sharding. Even with sharding, performance suffers due to serializing writes (and reads) through each shard’s leader and synchronizing with multiple leaders for cross-shard transactions.

ARE YOU OPEN SOURCE?

No. NuoDB is not open source. We do provide a Community Edition (CE) of our product that you can download today. To make things even easier, you can download our open source Helm Charts or Go Operator and get started in Kubernetes. You can also find our operator in Quay.io, AWS Marketplace, Google Cloud Platform, Red Hat Catalog, or OperatorHub.io. The CE is fully functional, but limits the scale-out to three Transaction Engines (TEs) and one Storage Manager (SM).

HOW EASY IS IT TO MIGRATE TO NUODB?

Any database migration requires work. NuoDB is not a fork of another database and as such, we implement our own SQL dialect. While we have added SQL that is compatible to Oracle and SQL Server dialect, you may be using certain vendor-specific extensions that may need to be modified.

More importantly, migrating existing applications to NuoDB might be easier than migrating to other distributed SQL databases because we abstract the complexity of sharding away and make the initial transition into the cloud and/or into Kubernetes easier. What does that mean? Existing applications that have been written for traditional single-node SQL databases generally are not easily partitionable or shardable. As a result, migrating to a distributed SQL database that requires partitioning or sharding may be very difficult for an existing application. NuoDB does support sharding, but you won’t need it from day 0. NuoDB Transaction Engines act as a form of LRU cache that dynamically adapts to the application access patterns without the need for large-scale application rewrites. If you have a monolithic application that is running against an enterprise database today, you can migrate to the cloud first and start breaking the monolith down second.

HOW DO YOU COMPARE TO COCKROACHDB/YUGABYTE?

While we are all distributed SQL databases, our architectural approach to achieve distribution is significantly different from the others, which results in different addressable use cases.

CockroachDB and Yugabyte both leverage automatic partitioning and a consensus algorithm to achieve distribution. In contrast, NuoDB’s architecture splits the compute and storage layers of a standard database into two different processes: Transaction Engines and Storage Managers. The Transaction Engines provide a form of LRU in-memory cache, which is coupled with Storage Managers for data persistence. Each layer can be scaled independently.

Our architectural approach allows our customers to migrate existing enterprise critical applications off Oracle, SQL Server, or DB2 more easily. As noted above, we don’t require a partitionable or shardable workload. Also, we don’t use a consensus algorithm for data management. As a result, we can address applications requiring very low latency and high transactional throughput. This was demonstrated in the Temenos benchmark, where we were able to deliver over 50K TPS for core banking transactions.

HOW DO I GET STARTED WITH NUODB?

Check out this straightforward tutorial, which will get you up and running with NuoDB Community Edition. If you have more questions, please contact us.

This article first appeared on the NuoDB technical blog.

Let me explain why all of these terms refer to essentially the same thing, although the ways that different databases achieve that scalability while maintaining consistency varies. As more and more companies started providing SaaS offerings that had no downtime, a pattern of pain points emerged:

• A monolithic database cannot provide the resiliency and high availability guarantees required by modern always-up, always-available applications.
• Scaling up is no longer viable. Machines big enough to run these types of workloads become too expensive, too quickly.
• NoSQL is not well suited for applications that require strong transactional and ACID guarantees.
• Explicit sharding is too complex and diverts engineering resources from what matters most: the business.

We’ve seen technologies come and go, but the same four fundamental motivations that are aligned with the shift to the cloud remain unchanged. NuoDB has been here since the beginning, and we continue to help our customers alleviate these pains.

MANAGING CONSISTENCY & ISOLATION

In ensuring that our database will meet the requirements of you, our customers, and the demands that your customers place on it, we’ve put a lot of thought into how we manage consistency and isolation and the isolation options that we provide. Our engineering team explored implementing a serializable isolation level but decided that the performance impact for our customers would be unacceptable. Serializable isolation is not useful for OLTP applications and is not widely required in the industry.

Serializable as an isolation level is a very useful academic concept, but the real-world OLTP applications that we are supporting use either Consistent Read (referred to as Repeatable Read in some literature) or predominantly Read Committed. The majority of well-known enterprise databases choose Read Committed as their default. Even MySQL, the biggest database that uses Repeatable Read as their default, requires configuration changes to use the serializable isolation level.

Solutions that are a thin sharding layer on top of another open-source database, such as PostgreSQL or MySQL, usually use the underlying Isolation Level of said database. Neither of these two databases offers serializable isolation by default. Since serializability is a niche feature, NuoDB decided to focus on pushing our performance to the limit while maintaining transactional and ACID guarantees.

FOCUSED ON PERFORMANCE

This focus on performance has paid off. NuoDB and Temenos recently reported a world record-breaking benchmark for core banking financial transactions. The benchmark in AWS demonstrated the ability to handle over half the world’s financial transactions in a single instance of Temenos running on NuoDB. During the benchmark, both Temenos and NuoDB were able to scale out to meet increasing load and then scale back down dynamically, demonstrating that financial customers can reduce TCO by using only the resources they need.

HANDLING FAULT TOLERANCE & HIGH AVAILABILITY REQUIREMENTS

NuoDB is a strongly consistent database, using replication between a user defined set of replicas. We recommend a replication factor of two to four, depending on the use case and deployment topology. In general, a system cannot be failure tolerant and highly available without some form of replication. By default, NuoDB allows the creation of databases without explicit sharding because our customers were already going through a lot of pain in the process of pursuing digital transformation goals and moving into the cloud. If needed, NuoDB can be explicitly sharded to improve performance, but more on that later. NuoDB delivers a scale-out and highly available database, without the additional complexities involved in adopting NoSQL for transactional applications or requiring sharding. These distributed SQL solutions enable companies to focus on what matters most: their business.

ADAPTABLE SCALE OUT OPTIONS

NuoDB has a two-tier system of Transaction Engines (TEs) and Storage Managers (SMs) to allow you to fine tune your environment to your particular needs. TEs act as a form of a data cache that dynamically adapts to the queries that are being executed on that TE. No complex, explicit, or manual sharding is required. TEs can be dynamically scale out or in to address changes in application demands. SMs, on the other hand, store all the data in their Storage Group on disk and guarantee durability. A user can decide how many SMs should serve which Storage Groups and hence decide on both the replication factor and the Availability vs. Persistence guarantees that their system requires. Storage Groups also allow users to partition the data across SMs, increasing IO throughput.

Our third tier, responsible for the administrative duties of the database, has been simplified over the years. This includes removing what was previously referred to as “Brokers” with a RAFT based Admin tier. The management of a distributed system is extremely complex and we provide a way to hide this complexity away behind an Admin API. This system works equally well in bare-metal and in orchestration platforms such as Kubernetes.

DEPLOYING IN MULTI-CLOUD, MULTI-CLUSTER ENVIRONMENTS

As more and more companies start their journey into the cloud, NuoDB is here to guide them and help them embrace an always-up, always-available mindset. At Kubecon 2019 in San Diego, we demonstrated a multi-cloud, multi-cluster deployment on Rancher Kubernetes Engine. This unique capability allows you to deploy a single logical database across multiple public or private clouds. Moving into production soon, Hong Kong FinTech leader WeLab is revolutionizing the banking experience for Hong Kong customers. (Stay tuned to learn more about how it works and why it is an important step for companies in their goal to avoid cloud vendor lock-in.)

To facilitate flexibility in deploying in hybrid environments and across multiple clouds, it’s essential that our database is fully functioning on whichever Cloud Service Provider (CSP) you choose. This is a key part of why NuoDB is cloud agnostic; to enable choice and flexibility, our distributed SQL database runs in all available public clouds, private clouds, and in hybrid cloud environments. Our continuous integration pipelines test and validate the product in Amazon, Azure, and Google Cloud Platform. We also work with AKS, EKS, GKE, and Rancher. Of course not everyone is using us with a CSP or in Kubernetes, so we also work on bare metal and any major VM infrastructure. Deployment in a public cloud is not required and you can easily deploy NuoDB on-prem and in your private homegrown clouds, which allows you the choice to deploy when, where, and how you want.

DELIVERING A SCALABLE, RESILIENT SQL DATABASE

The concept behind NuoDB has been around since 2008, based on Jim Starkey’s view that traditional SQL databases don’t scale well beyond a single system. This was well before the term NewSQL was coined. In 2020, we have 12 years of experience with building a resilient and ACID compliant database. And as cloud computing adoption has accelerated, our focus hasn’t needed to change. We’ve been building a database that thrives in distributed deployment environments since day one. Naturally, as technology evolves, we continually work to make sure that NuoDB is cloud native, works with microservices, containers, and container orchestration platforms. We explored Docker Swarm and Mesos when they were still a competing option, but decided to focus purely on Kubernetes once it became the leading player. We support Kubernetes Helm Charts and are also actively working on a Kubernetes Operator, which is currently available in our Community Edition, freely available for you to try on the Kubernetes Marketplace of your choice.

Many of the other NewSQL solutions took a shortcut and simply used the SQL layer and storage engines of other open source products. These are fundamentally limited in how far they can grow and how optimized they can become. NuoDB has been working on its own SQL and storage solutions for 12 years, which are highly optimized for our particular architecture running in the cloud. Our SQL engine is not limited by a 3rd party upstream product and has virtually unlimited growth potential that allows us to adapt to the dynamic and ever changing environment we are in.

NuoDB has come a long way since our 1.0 release seven years ago. We have been battle-hardened in production and we are very proud of the achievements we have made since then. We believe in our multi-tiered SM/TE architecture, which is adaptable for the various workloads that our customers require. We believe in our caching mechanism that allows TEs to only access a subset of the data, dynamically adapting to their current workload without the need to shard their data and workloads manually. We believe in our mission to make the use of distributed SQL easier and to support you in your journey from a monolithic database into the cloud. We believe that an important feature of any modern database is that it must be always up and always available. Resilience to failure has been a fundamental design principle in our architecture and day-to-day development. The statement also validates what we have been observing: the core of NuoDB was designed right and is well suited for the shift to the cloud. This means that we have an incredible head start with a proven solution.

This article first appeared on the NuoDB technical blog.

EXCLUSIVE access is required for DDL that visits all records (various index operations for example) and distributed concurrent access is not possible. Exclusive access can also be used by operations that need to work around MVCC write skew anomalies.

SHARED locks need to be fast, consume as little memory as possible, involve no additional nodes in the cluster, and cause no additional messaging.

EXCLUSIVE locks, on the other hand, need to pay the cost.

MVCC uses row locks to serialise updates to a single record. Both transactional locks and row locks participate in the same deadlock detection process, but are otherwise independent. The following table explains the interaction between transactional locks and MVCC row locks. If you are interested in how row locks work, I recommend reading our MVCC blog series.

USE OF TRANSACTIONAL LOCKS

Transactional locks are tied to the lifetime of a transaction. A lock can not exist without a transaction and it can not be released while the transaction still exists. The only way to release a transactional lock, is to resolve the transaction. There are currently three ways to resolve a transaction: commit, rollback, and failure (local or cluster wide).

The lock is held until all effects of the transaction have been made visible to another node. If a transaction TX is connected to TE T, it does not need to wait for the resolution of the lock if it does not modify any locked resource.

Similarly to a programming language mutex, once a SHARED or EXCLUSIVE lock has been acquired, the transaction needs to verify that no change happened since the beginning of the transaction. This is simple for READ COMMITTED transactions, since they do not guarantee a consistent snapshot across statements. An RC transaction will acquire all required SHARED locks, wait for any EXCLUSIVE locks that might have been acquired in the past and their effects; only then does it freeze it’s visibility snapshot. This ordering guarantees that the statement will see the newest, correct, and consistent snapshot of the database.

But what about CONSISTENT READ transactions? When using CR, the snapshot is frozen in time and can not be updated. This means that if a CR encounters an EXCLUSIVE lock, it might need to abort. If the EXCLUSIVE operation committed, this might mean that the view of the database is no longer compatible with the CR transaction. If it rolled back or failed, the CR can continue. If a CR can not continue you will see the exception ‘Table has been changed’. NuoDB only aborts transactions that have been proven to contain conflicting updates.

As I explained above, a transactional lock is similar to a mutex and should be treated that way. When using them, we recommend the check;lock;check pattern. Due to the tricky nature of snapshot visibility, NuoDB prohibits the use of the LOCK statement in CONSISTENT READ transactions. Since both check statements in the check;lock;check pattern would be executed with the same snapshot, they are guaranteed to return the same results. Any update that happened strictly before the EXCLUSIVE access was granted to the resource would not be reflected in the second check. To protect the application developer from such a mistake, NuoDB does not allow the LOCK statement in consistent read isolation level.

USING LOCKS TO PREVENT WRITE SKEW

Write skew is a well known anomaly in MVCC that we might cover in future articles. For a quick primer…

A table contains three employees Bob, Mary, and Sue with their respective salaries:

Since the company had a good quarter, the CEO now opens a new req AND approves a salary increase for all the existing employees. The total cost can not be higher than 500.

The department head of department D1 looks at the total salary, calculates how much more the employees can be payed, and divides the difference equally. In SQL, that would look like:

SQL> select sum(salary) as “Existing Salaries” from employees;

Existing Salaries
——————

    320    SQL**\>** update employees **set** salary = salary + **(****select** **(**500-sum**(**salary**)****)****/**count**(**salary**)** from employees**)** ; SQL**\>** **select** **\*** from employees;

NAME SALARY
—– ——-

Bob 160
Mary 210
Sue 130

SQL> select sum(salary) from employees;

SUM
—-

500

Concurrently, the department head of D2 adds a new employee:

SQL> select sum(salary) from employees;

SUM
—-

320

SQL> insert into employees values(‘Chung’, 500-(select sum(salary) from employees)); SQL> select * from employees;

NAME SALARY
—– ——-

Bob 100
Mary 150
Sue 70
Chung 180

SQL> select sum(salary) from employees;

SUM
—-

500

As we can see, these two transactions do not conflict in MVCC. Once both transactions commit, the CEOs limit is violated.

SQL> select sum(salary) from employees;

SUM
—-

680

To prevent this from happening, one of the transactions can acquire an EXCLUSIVE lock. Let us look at the second transaction with locking. We will be using the check;lock;check pattern as explained in the introduction.

SQL> start transaction isolation level read committed; SQL> select sum(salary) from employees;

SUM
—-

320

SQL> lock table employees; // blocks until T1 resolves SQL> select sum(salary) from employees;

SUM
—-

500
SQL> // Chung can not be hired

UNDER THE HOOD

So how is this logic implemented? NuoDB does not use majority consensus algorithms, such as Paxos or Raft; instead, NuoDB depends on Chairmanship. All EXCLUSIVE requests will need to be granted by the chairman to ensure strict ordering. If an EXCLUSIVE request gets denied, the transaction will have to wait for the current holder to resolve before it can retry.

EXCLUSIVE locks use a form of INTENT. The chairman broadcasts an INTENT to acquire an EXCLUSIVE lock. All other engines will reply with a collection of all current SHARED locks.

Once all SHARED locks have been resolved, the INTENT is promoted to an EXCLUSIVE lock. Once an INTENT has been placed, no further SHARED or EXCLUSIVE locks can be placed. The promotion from INTENT to EXCLUSIVE is done without any additional messaging.

Figure 1. Lifecycle and messaging involved in Transactional Locks

The reliable broadcast protocol that NuoDB uses guarantees that the INTENT is either placed everywhere or nowhere. If the chairman dies during the locking protocol, the requestor will wait for the election of a new chairman before retrying.

Due to the use of INTENT-like locks, it is impossible to starve an EXCLUSIVE requestor. On the other hand, an incomplete EXCLUSIVE request will block future SHARED locks long before the requestor can take advantage of the lock. NuoDB assumes that an EXCLUSIVE lock is a rare operation and that application developers understand the cost of a distributed cluster-wide exclusive access. Long running SHARED lock owners can lock out everybody else from the resource for a long period of time. If you notice that taking an EXCLUSIVE lock takes a long period of time, consider consulting NuoDB pseudo table system.transactionallocks for debugging information.

Figure 2. Long running SHARED lock prevents any action on the table

In Figure 2 above we can see a situation when a long running analytical query by USER 1 owns a SHARED lock on table T1. While that transaction is in progress, no EXCLUSIVE lock on that resource can be acquired. When a DBA attempts to alter table T1 (which requires EXCLUSIVE access), her transaction will place an INTENT and block on the existing long running transaction. Once an INTENT has been placed, no further SHARED locks can be acquired. All transactions that already own a lock are allowed to proceed, but no new transactions can acquire the lock until the INTENT has been resolved. In this scenario, we can see that USER 2 is unable to insert into the table.

Here is the information that you can acquire from system tables in the situation described in Figure 2.

SQL> select * from system.transactionallocks;

OBJECTID TRANSID NODEID LOCKTYPE SOURCENODE
——— ——– ——- ——— ———–

72       1410      2    Exclusive      2      
72       1026      2    Shared         2       SQL**\>** **select** **id**,state, blockedby from system.transactions;

ID STATE BLOCKEDBY
—- —— ———-

1026 Active -1
1410 Active 1026
1666 Active 1410

To let the system proceed, one of the actors in the dependency graph will need to resolve their transaction (commit, roll back, or fail).

WHAT IS THE COST OF A LOCK?

The following section assumes that there are no conflicts - that is none of the requests hit a conflicting lock that is already in place. If there is a conflict, the lock request will have to wait for the resolution of the lock, and hence the transaction that owns that lock. Since the lifetime of any arbitrary transaction is outside the control of NuoDB, we will take that out of the equation.

Acquiring a SHARED lock (done automatically by NuoDB) is zero overhead. We have verified this by various in-house performance benchmarks, TPC-C, and YCSB.

The cost of acquiring an EXCLUSIVE lock is three times the network latency.

• Requestor -> Chairman (network latency in ms)
• Chairman -> all other nodes (network latency in ms)
• All other nodes -> requestor (latency in ms)

Since the requestor needs to receive a grant from every single TE in the system, the network latency of TEs located in remote data centers can be the major decisive component.

Figure 3. Network latency and cost of a lock

Each lock is held until the end of its transaction. If a transaction contains multiple statements working on different resources, all those resources will be locked. Keep this in mind when you are performing large schema changes.

SUMMARY

We have learned that expensive EXCLUSIVE transactional locks are not being used for normal CRUD operations. A CUD operation can operate without explicit approval from all nodes in the cluster by using a SHARED lock. An EXCLUSIVE lock is only required for certain metadata modifications and for special application-level use cases.

You can acquire an EXCLUSIVE lock for any type of operation that requires exclusive access to your table. It is possible to use EXCLUSIVE lock to work around write skew anomalies, but the recommended approach is to cause a write-write conflict by adding an additional void update on a well known record.

NuoDB does not use consensus algorithm, but instead depends on chairmanship for lock resolution.

This article first appeared at the NuoDB Tech Blog under the name Distributed Transactional Locks

One transaction increases the salary of everyone in a department by 10%; another transaction inserts a new employee with a salary X. Since the two transactions do not conflict, MVCC does not prevent either of them from committing. After both resolve, the overall salary in the department could be above the budget. To prevent a similar situation, the application developer might want to have exclusive access to the table.

NuoDB 3.2.2 exposes a new type of lock that guarantees exclusive access to a resource across the distributed cluster. We call them Transactional Locks and they are the underlying mechanism powering the new NuoDB LOCK statement.

ZERO NORMAL CASE OVERHEAD

NuoDB is a distributed database. When we were designing Transactional Locks, our primary concern was the happy path - the DML that you execute thousands of times a second, does not pay the cost of a distributed operation.

Read DML transactions (select) will never acquire any lock. The consistency is guaranteed by Multi Version Concurrency Control. A Consistent Read transaction containing multiple select statements keeps a consistent metadata model of the database regardless of concurrent schema modifications.

Write DML transactions (update, insert, delete, select for update) acquire SHARED locks. These locks are local to the Transaction Engine that is running the transaction. Unless there is an EXCLUSIVE lock already acquired, acquiring this lock will always succeed and does not conflict with any other read or write DML operation. These SHARED locks are not distributed, serialized, communicated, or replicated throughout the cluster. Truly zero overhead.

PAYING THE COST: TRANSACTIONS THAT REQUIRE UNIQUE ACCESS

Nothing in life comes for free, and neither does distributed consistency. Since SHARED locks are not known throughout the cluster, any operation that needs EXCLUSIVE access to a resource needs to pay the cost.

A operation, which requires EXCLUSIVE access to the altered resource, needs to wait for all transactions on all Transaction Engines with SHARED locks to finish. Once all transactions have been resolved (committed or rolled back), the requestor has exclusive access.

Transactional Locks make sure that the requesting operation eventually acquires the EXCLUSIVE lock, avoiding any starvation. As a NuoDB application developer, you should make sure that there are no long running write DML statements in flight. The exclusive requestor will have to wait for any long running transactions, possibly locking out other writers. Long running analytical SELECT queries will not conflict with transactional locks.

LOCK SQL STATEMENT

All write DML acquires SHARED locks automatically and without any user intervention. As of 3.2.2, no EXCLUSIVE locks are acquired automatically.

As described in the introduction of this article, there might be situations when your application wants to have unique access to a resource. We offer you the LOCK statementthat does exactly that. Just keep in mind that anything that is exclusive and/or unique in a distributed system can have impact on the whole cluster.

When using Transactional Locks, NuoDB recommends the following:

• Make sure you have AUTOCOMMIT OFF or that you started a new transaction via START TRANSACTION.
• Your transaction uses READ COMMITTED. Consistent read transactions can not see all changes that happened after a snapshot was taken. This can result in lots of update conflicts on the locking transaction that will prevent you from changing the metadata.
• Do not have long running DML transactions.

We do not offer the ability to manually lock a resource in SHARED mode. All write DML acquires a SHARED lock. To prevent the acquisition of a EXCLUSIVE LOCK on a table, NuoDB recommends executing a dummy update update t set i = i where 0 = 1;

THERE IS NO UNLOCK

Transactional locks cannot be released voluntarily. The only way to unlock a resource is to commit or roll back the transaction.

Once obtained, the lock is held for the remainder of the current transaction. There is no UNLOCK TABLE command; locks are always released at the transaction end.

MORE READING

We will explore the technical fundamentals of transactional locks in future articles.

This article first appeared at the NuoDB Tech Blog under the name How Transaction Locks support Zero Overhead Distributed DML