Case Study · Real Estate / ERP Recovery & Managed Services

Not one business day lost: recovering a developer’s core Oracle EBS over a bank holiday

An unplanned loss of the production environment struck a leading Egyptian real-estate developer’s core Oracle E-Business Suite early on a bank holiday morning. Khwarizm rebuilt and restored the estate across the holiday weekend — with zero data loss, and in production before the next business day opened — and has operated the ERP and its reporting layer, largely remotely, ever since.

A property developer’s business runs on commitments made years in advance. A unit sold in one quarter is delivered several quarters later, and the instalment schedule behind it runs for a decade. All of it lives in the ERP. When a leading Egyptian real-estate developer’s core Oracle E-Business Suite went out of service in an unplanned loss of the production environment, the company did not lose a system. It lost the ability to see what it owed, what it was owed, and what it had already collected. Khwarizm’s phone rang early on a bank holiday morning.

What a developer loses when the ledger goes dark

The client is one of Egypt’s largest real-estate developers, administering a vast land bank and a development backlog — units sold but not yet handed over — that runs to many billions of Egyptian pounds. A backlog of that size is a promise, administered almost entirely in software: instalment schedules against thousands of buyers, contractor invoices paid against a construction programme, a general ledger that has to close every month whatever else is happening.

None of that stopped being owed when the system stopped. It only stopped being visible: money kept falling due from buyers and being committed to contractors, against a record nobody could read. That is the particular cruelty of an ERP outage: unlike a factory line, nothing outside the building appears to halt, so the damage accumulates quietly — in receivables, in payables, in a period-end getting closer every day.

The call came on a holiday morning

Timing of that kind cuts both ways. A bank holiday running into a weekend is the largest uninterrupted recovery window a business ever gets: no postings accumulating to be reconciled, no users to lock out of a half-rebuilt system. It is also when the fewest people are reachable. Khwarizm mobilised on the customer’s call and worked the holiday through, day and night, before any contract existed — a disaster does not wait for a purchase order to clear.

The estate was not fragile to begin with: Oracle E-Business Suite R12.1 on Oracle Database 12c, on Linux, high-availability across both the application and database tiers, carrying the Financials modules. But high availability protects against a component dying. It does nothing about an event that corrupts what the healthy nodes are faithfully replicating — which is what separates a corruption event from a simple outage, and why recovery here was two questions rather than one. Can the data be brought back, and can the environment it lands in be trusted afterwards.

What carried the recovery was preparation of a different kind. The client had treated recovery time and recovery point as targets to design for, not assumptions to discover mid-incident. A recovery that loses nothing is not luck — it is what a backup regime looks like when someone has decided in advance how much data the business can afford to lose, and tested that the answer holds.

Khwarizm sequenced the work as rebuild-and-restore-forward — establish a known-good state, stand the environment up, bring the data forward onto it, then validate against real business processes, a receipt applied to an instalment or a payables run, not against a checklist of services responding on the right ports. The core Oracle E-Business Suite was running in production before the next business day opened, with zero data loss. Nothing was rekeyed from paper and no balance had to be argued out with a buyer or a contractor: the recovered system carried every transaction the original held.

The reporting layer had to come back too

Reporting is the half of an ERP recovery that gets deferred, because on day one nobody escalates a missing dashboard. Then the first period-end arrives, and a transaction system nobody can query turns out to be a filing cabinet. The developer’s reporting ran on Oracle BI (OBIEE) alongside Oracle Discoverer and BI Publisher — all dependent on the estate that had gone down.

Khwarizm recovered the reporting layer inside the same window, not as a follow-on phase. The consequence is narrow and worth stating plainly: the first period-end after the incident closed on the system, not in spreadsheets reconciled backwards afterwards.

Then somebody has to run it

Recovery projects end; the exposure that produced them does not. And the calendar that helped here will not help twice — a Tuesday morning would have cost business hours no amount of speed gives back. That is the argument for keeping the people who rebuilt an estate close to it: when there is no holiday to work through, the response starts from familiarity rather than discovery.

The client asked Khwarizm to stay on and operate it — the whole estate, not a slice. Day-to-day administration of the Oracle E-Business Suite and its database, patching and release management, performance work, period-end support, incident handling, and the OBIEE and BI Publisher reporting layer, all run by the same team. That last point matters more than it sounds: when a number on a report is wrong, the argument over whether it is a reporting fault or a posting fault is where most managed-service time goes. Here there is nobody to have it with.

Most of that has been delivered remotely, by design rather than as a concession. It buys continuity of the same named engineers instead of whoever is on site that week. What it demands in exchange is unambiguous escalation, disciplined change control, and access agreed in advance rather than mid-incident — which matters far more after an event like this than before one. On-site presence is reserved for work that needs hands in the room. The estate has run at 99.9% availability since, which is the only fair test of a distance model.

The results

Most of the company came back to a system that behaved exactly as it had the week before — the strangest measure of success in this work, where the best outcome is the one nobody notices. A buyer’s instalment is applied the day it is received. The ledger closes at month-end, without anyone checking whether the record survived. For a company carrying many billions of pounds in promises to people who have paid for homes not yet built, that ordinariness is the whole point.