Article
How to Choose the Right Integration Method | MAbbaz.com
mabbaz.com
Quoted on this wiki
Every place a page here uses this source, in the order the words come in it.
“Almost every integration mistake I have been called in to unwind started the same way: someone picked the method first and matched it to the problem afterwards.” Start with the pillar: this article assumes you already understand the landscape of integration patterns and terminology. If you want the full map first, read Enterprise System Integration Explained, then come back here to make the actual choice. The pillar tells you what the methods are; this guide tells you how to pick between them.
Direction: who initiates, and does data flow one way or both? Does the source push when something changes, or does the target pull on a schedule? Is it a one-way feed or a two-way sync? Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs. Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method. “Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change.” Skills: what can your team actually operate? The most elegant streaming architecture is worthless if nobody on the team can run a broker at 3am when it stops consuming. Match the method to the skills you have or can realistically acquire. A small team with strong SQL and no platform engineers is better served by well-built batch ETL than by a distributed event mesh they cannot support. Sustainability beats sophistication every time.
Direction: who initiates, and does data flow one way or both? Does the source push when something changes, or does the target pull on a schedule? Is it a one-way feed or a two-way sync? Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs. Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method. “The partner constraint often overrides every other consideration, so establish it early before you design anything.” Skills: what can your team actually operate? The most elegant streaming architecture is worthless if nobody on the team can run a broker at 3am when it stops consuming. Match the method to the skills you have or can realistically acquire. A small team with strong SQL and no platform engineers is better served by well-built batch ETL than by a distributed event mesh they cannot support. Sustainability beats sophistication every time.
2. The questions that decide it “Every method choice I make runs through the same five questions, in roughly this order.” Timing: how fresh does the data need to be? Does the receiving system need the data within seconds, or is within hours acceptable? A fraud check needs the transaction now. A monthly management report does not care whether the numbers arrived at 2am or 4am. Timing is the single biggest determinant of method, because it separates the real-time family (APIs, webhooks, streaming) from the scheduled family (batch, ETL, file transfer). Be ruthlessly honest here, because "real time" is the most over-specified requirement in integration. Most things people call real time are comfortably served by a five-minute or hourly cycle.
Every method choice I make runs through the same five questions, in roughly this order. None of them require deep technical knowledge to answer. They require you to be honest about what the business actually needs, which is usually harder. “Timing is the single biggest determinant of method, because it separates the real-time family (APIs, webhooks, streaming) from the scheduled family (batch, ETL, file transfer).” Volume: how much data moves, and how often? Are you moving one record when a customer clicks save, or five million rows every night? Small, frequent, transactional payloads favour APIs and events. Large, periodic bulk movement favours batch and ETL. Trying to push a five-million-row load through a per-record API is one of the most common and most painful anti-patterns I see, and it usually surfaces as a source system falling over under the request load.
Every method choice I make runs through the same five questions, in roughly this order. None of them require deep technical knowledge to answer. They require you to be honest about what the business actually needs, which is usually harder. “Most things people call real time are comfortably served by a five-minute or hourly cycle.” Volume: how much data moves, and how often? Are you moving one record when a customer clicks save, or five million rows every night? Small, frequent, transactional payloads favour APIs and events. Large, periodic bulk movement favours batch and ETL. Trying to push a five-million-row load through a per-record API is one of the most common and most painful anti-patterns I see, and it usually surfaces as a source system falling over under the request load.
Timing: how fresh does the data need to be? Does the receiving system need the data within seconds, or is within hours acceptable? A fraud check needs the transaction now. A monthly management report does not care whether the numbers arrived at 2am or 4am. Timing is the single biggest determinant of method, because it separates the real-time family (APIs, webhooks, streaming) from the scheduled family (batch, ETL, file transfer). Be ruthlessly honest here, because "real time" is the most over-specified requirement in integration. Most things people call real time are comfortably served by a five-minute or hourly cycle. “Small, frequent, transactional payloads favour APIs and events. Large, periodic bulk movement favours batch and ETL.” Direction: who initiates, and does data flow one way or both? Does the source push when something changes, or does the target pull on a schedule? Is it a one-way feed or a two-way sync? Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs. Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method.
Volume: how much data moves, and how often? Are you moving one record when a customer clicks save, or five million rows every night? Small, frequent, transactional payloads favour APIs and events. Large, periodic bulk movement favours batch and ETL. Trying to push a five-million-row load through a per-record API is one of the most common and most painful anti-patterns I see, and it usually surfaces as a source system falling over under the request load. “Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs.” Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything.
Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything. “Match the method to the skills you have or can realistically acquire.” Answer those five and you have narrowed a dozen options down to two or three. The decision flow below turns the answers into a recommendation. For the deeper trade-off between the two big timing families, the companion guide on batch versus real-time integration covers that split in detail.
Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything. “A small team with strong SQL and no platform engineers is better served by well-built batch ETL than by a distributed event mesh they cannot support.” Answer those five and you have narrowed a dozen options down to two or three. The decision flow below turns the answers into a recommendation. For the deeper trade-off between the two big timing families, the companion guide on batch versus real-time integration covers that split in detail.
Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything. “Sustainability beats sophistication every time.” Answer those five and you have narrowed a dozen options down to two or three. The decision flow below turns the answers into a recommendation. For the deeper trade-off between the two big timing families, the companion guide on batch versus real-time integration covers that split in detail.
The first thing to unlearn is the idea that one integration style is superior. Consultants and vendors have strong incentives to sell a house style, whether that is an API-first mandate, an event-driven architecture, or an integration platform subscription. In real environments, a healthy landscape almost always runs several methods at once. The finance team pulls a nightly general-ledger extract as a batch file. The web storefront calls an inventory API in real time. The warehouse emits an event when a shipment leaves the dock. A B2B partner exchanges purchase orders over EDI because that is the only thing their fifteen-year-old ERP speaks. All four are correct, because each is matched to a different combination of timing, volume, direction and partner constraint. “Force a real-time API onto a bulk data-migration job and you get a chatty, slow, rate-limited nightmare that hammers the source system.” 2. The questions that decide it
The first thing to unlearn is the idea that one integration style is superior. Consultants and vendors have strong incentives to sell a house style, whether that is an API-first mandate, an event-driven architecture, or an integration platform subscription. In real environments, a healthy landscape almost always runs several methods at once. The finance team pulls a nightly general-ledger extract as a batch file. The web storefront calls an inventory API in real time. The warehouse emits an event when a shipment leaves the dock. A B2B partner exchanges purchase orders over EDI because that is the only thing their fifteen-year-old ERP speaks. All four are correct, because each is matched to a different combination of timing, volume, direction and partner constraint. “Force a nightly batch onto a customer-facing stock check and you get sold-out items still showing as available for twenty hours.” 2. The questions that decide it
Volume: how much data moves, and how often? Are you moving one record when a customer clicks save, or five million rows every night? Small, frequent, transactional payloads favour APIs and events. Large, periodic bulk movement favours batch and ETL. Trying to push a five-million-row load through a per-record API is one of the most common and most painful anti-patterns I see, and it usually surfaces as a source system falling over under the request load. “Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method.” Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything.
Direction: who initiates, and does data flow one way or both? Does the source push when something changes, or does the target pull on a schedule? Is it a one-way feed or a two-way sync? Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs. Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method. “Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change.” Skills: what can your team actually operate? The most elegant streaming architecture is worthless if nobody on the team can run a broker at 3am when it stops consuming. Match the method to the skills you have or can realistically acquire. A small team with strong SQL and no platform engineers is better served by well-built batch ETL than by a distributed event mesh they cannot support. Sustainability beats sophistication every time.
Direction: who initiates, and does data flow one way or both? Does the source push when something changes, or does the target pull on a schedule? Is it a one-way feed or a two-way sync? Push-on-change points you at webhooks and events. Pull-on-schedule points you at batch and polling APIs. Two-way sync, where both systems can change the same record, is the hardest case and needs deliberate conflict handling regardless of method. “The partner constraint often overrides every other consideration, so establish it early before you design anything.” Skills: what can your team actually operate? The most elegant streaming architecture is worthless if nobody on the team can run a broker at 3am when it stops consuming. Match the method to the skills you have or can realistically acquire. A small team with strong SQL and no platform engineers is better served by well-built batch ETL than by a distributed event mesh they cannot support. Sustainability beats sophistication every time.
Every method above can be hand-built with code, or assembled on an integration platform as a service (iPaaS) that provides pre-built connectors, visual flow design, hosting and monitoring out of the box. This is an orthogonal choice: iPaaS is not a method, it is a way to deliver the methods. The decision hinges on connector coverage, team capacity and the number of integrations you expect to run. Lean toward iPaaS when you have many integrations to build across popular SaaS applications, when a small team needs to move quickly without owning integration infrastructure, and when the platform already has connectors for your systems so you assemble rather than code. The productivity gain is real, and so is the reduction in operational burden. “Many mature landscapes end up with both: an iPaaS handling the long tail of SaaS-to-SaaS connections, and hand-built integrations for the few core, high-volume, differentiated flows.” 9. References
Final thoughts “Ask how fresh the data must be, how much of it moves and how often, who initiates and in which direction, whether the other side is a partner you must conform to, and what your team can actually operate.” When two methods still tie, let operations break the tie: pick the one that is cheaper to run and easier to support when it fails at night. A landscape built that way tends to be quiet, and quiet is the real mark of good integration. If you are staring at a decision like this on a specific interface and want a second opinion grounded in having made the choice many times, that is exactly the kind of conversation I am glad to have.
Every method above can be hand-built with code, or assembled on an integration platform as a service (iPaaS) that provides pre-built connectors, visual flow design, hosting and monitoring out of the box. This is an orthogonal choice: iPaaS is not a method, it is a way to deliver the methods. The decision hinges on connector coverage, team capacity and the number of integrations you expect to run. Lean toward iPaaS when you have many integrations to build across popular SaaS applications, when a small team needs to move quickly without owning integration infrastructure, and when the platform already has connectors for your systems so you assemble rather than code. The productivity gain is real, and so is the reduction in operational burden. “Lean toward building it yourself when the integration is few in number but deep and specific, when it sits in a performance-critical or highly regulated path, when your systems are bespoke enough that no connector exists, or when platform subscription and per-transaction costs would outweigh the engineering saved.” 9. References
Does it need to be near real-time? yes Source pushes on a change? yes Webhook event notification no High volume of continuous events? yes Messaging / Streaming no API (request / response) no (hours are fine) External trading partner? yes Standard B2B document format? yes EDI B2B trading documents no File transfer CSV / XML over SFTP no (internal) ETL / Batch bulk scheduled load “When you find yourself between two boxes, fall back to the two tie-breakers that matter most in operations: which method is cheaper to run, and which one your team can support at 3am.” 4. The selection matrix
Partner: is the other side internal or an external trading partner? Integrating two systems you control gives you free choice of method. Integrating with an external partner, a bank, a supplier, a government portal, a logistics provider, means you take whatever they support, and that is frequently EDI, a fixed file format, or a specific API contract you cannot change. The partner constraint often overrides every other consideration, so establish it early before you design anything. “Match the method to the skills you have or can realistically acquire.” Answer those five and you have narrowed a dozen options down to two or three. The decision flow below turns the answers into a recommendation. For the deeper trade-off between the two big timing families, the companion guide on batch versus real-time integration covers that split in detail.